From 658f2da1f33aaf86d7b345891f479462a7d1e2d3 Mon Sep 17 00:00:00 2001 From: Thomas Huth Date: Fri, 27 May 2022 16:35:31 +0100 Subject: [PATCH 01/33] .gitlab-ci.d/container-cross: Fix RISC-V container dependencies / stages MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The "riscv64-debian-cross-container" job does not depend on any other container job from the first stage, so we can move it to the first stage, too. The "riscv64-debian-test-cross-container" job needs the debian11 container, so we should add a proper "needs:" statement here. Signed-off-by: Thomas Huth Message-Id: <20220524093141.91012-1-thuth@redhat.com> Signed-off-by: Alex Bennée Reviewed-by: Alistair Francis Message-Id: <20220527153603.887929-2-alex.bennee@linaro.org> --- .gitlab-ci.d/container-cross.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml index e622ac2d21..ac15fce9b6 100644 --- a/.gitlab-ci.d/container-cross.yml +++ b/.gitlab-ci.d/container-cross.yml @@ -125,7 +125,7 @@ ppc64el-debian-cross-container: riscv64-debian-cross-container: extends: .container_job_template - stage: containers-layer2 + stage: containers # as we are currently based on 'sid/unstable' we may break so... allow_failure: true variables: @@ -135,6 +135,7 @@ riscv64-debian-cross-container: riscv64-debian-test-cross-container: extends: .container_job_template stage: containers-layer2 + needs: ['amd64-debian11-container'] variables: NAME: debian-riscv64-test-cross From 6fd4e75610c08820faabf9c544f2f2af2b1b8334 Mon Sep 17 00:00:00 2001 From: Thomas Huth Date: Fri, 27 May 2022 16:35:32 +0100 Subject: [PATCH 02/33] .gitlab-ci.d/crossbuilds: Fix the dependency of the cross-i386-tci job MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The cross-i386-tci job uses the fedora-i386-cross image, so we should make sure that the corresponding job that builds it (the i386-fedora-cross-container job) has finished before we start the TCI job. Signed-off-by: Thomas Huth Reviewed-by: Richard Henderson Message-Id: <20220524092600.89997-1-thuth@redhat.com> Signed-off-by: Alex Bennée Message-Id: <20220527153603.887929-3-alex.bennee@linaro.org> --- .gitlab-ci.d/crossbuilds.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml index 17d6cb3e45..4a5fb6ea2a 100644 --- a/.gitlab-ci.d/crossbuilds.yml +++ b/.gitlab-ci.d/crossbuilds.yml @@ -62,6 +62,8 @@ cross-i386-user: cross-i386-tci: extends: .cross_accel_build_job timeout: 60m + needs: + job: i386-fedora-cross-container variables: IMAGE: fedora-i386-cross ACCEL: tcg-interpreter From 861dca3707a73cbe022cefe52f7224b28350316b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc-Andr=C3=A9=20Lureau?= Date: Fri, 27 May 2022 16:35:33 +0100 Subject: [PATCH 03/33] gitlab-ci: add meson JUnit test result into report MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This allows the gitlab UI to show the test results in different ways, see doc: https://docs.gitlab.com/ee/ci/unit_test_reports.html#how-it-works Previous we only reports avocado test results (.avocado_test_job_template), with this change, the qemu/meson tests are also covered. Signed-off-by: Marc-André Lureau Message-Id: <20220525173411.612224-1-marcandre.lureau@redhat.com> [AJB: expand the commit description] Signed-off-by: Alex Bennée Reviewed-by: Thomas Huth Message-Id: <20220527153603.887929-4-alex.bennee@linaro.org> --- .gitlab-ci.d/buildtest-template.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml index dc6d67aacf..b381345dbc 100644 --- a/.gitlab-ci.d/buildtest-template.yml +++ b/.gitlab-ci.d/buildtest-template.yml @@ -44,6 +44,8 @@ expire_in: 7 days paths: - build/meson-logs/testlog.txt + reports: + junit: build/meson-logs/testlog.junit.xml .avocado_test_job_template: extends: .common_test_job_template From 85b141ea6ae448b718d438d9cf0009d73116f8d1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alex=20Benn=C3=A9e?= Date: Fri, 27 May 2022 16:35:34 +0100 Subject: [PATCH 04/33] meson.build: fix summary display of test compilers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The recent refactoring of configure.sh dropped a number of variables we relied on for printing out information. Make it simpler. Fixes: eebf199c09 (tests/tcg: invoke Makefile.target directly from QEMU's makefile) Signed-off-by: Alex Bennée Reviewed-by: Richard Henderson Message-Id: <20220527153603.887929-5-alex.bennee@linaro.org> --- meson.build | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/meson.build b/meson.build index bf318d9cbb..bc6234c85e 100644 --- a/meson.build +++ b/meson.build @@ -3735,12 +3735,8 @@ foreach target: target_dirs config_cross_tcg = keyval.load(tcg_mak) target = config_cross_tcg['TARGET_NAME'] compiler = '' - if 'DOCKER_CROSS_CC_GUEST' in config_cross_tcg - summary_info += {target + ' tests': config_cross_tcg['DOCKER_CROSS_CC_GUEST'] + - ' via ' + config_cross_tcg['DOCKER_IMAGE']} - elif 'CROSS_CC_GUEST' in config_cross_tcg - summary_info += {target + ' tests' - : config_cross_tcg['CROSS_CC_GUEST'] } + if 'CC' in config_cross_tcg + summary_info += {target + ' tests': config_cross_tcg['CC']} endif endif endforeach From 06885cf93556adc3fd67aad0c279a0deb400bb7d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alex=20Benn=C3=A9e?= Date: Fri, 27 May 2022 16:35:35 +0100 Subject: [PATCH 05/33] tests/lcitool: fix up indentation to correct style MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 3 space indentation snuck into the initial commit. Clean it up before we let it get established. I've also: - removed unused os import - added double lines between functions - added some comments and grouped and sorted the generation stanzas My lint tool is also recommending using f-strings but that requires python 3.6. Signed-off-by: Alex Bennée Cc: Daniel P. Berrangé Reviewed-by: Daniel P. Berrangé Message-Id: <20220527153603.887929-6-alex.bennee@linaro.org> --- tests/lcitool/refresh | 128 ++++++++++++++++++++++++------------------ 1 file changed, 73 insertions(+), 55 deletions(-) diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh index fb49bbc441..dc1fc21ef9 100755 --- a/tests/lcitool/refresh +++ b/tests/lcitool/refresh @@ -13,14 +13,13 @@ # the top-level directory. import sys -import os import subprocess from pathlib import Path if len(sys.argv) != 1: - print("syntax: %s" % sys.argv[0], file=sys.stderr) - sys.exit(1) + print("syntax: %s" % sys.argv[0], file=sys.stderr) + sys.exit(1) self_dir = Path(__file__).parent src_dir = self_dir.parent.parent @@ -30,76 +29,95 @@ lcitool_path = Path(self_dir, "libvirt-ci", "lcitool") lcitool_cmd = [lcitool_path, "--data-dir", self_dir] + def atomic_write(filename, content): - tmp = filename.with_suffix(filename.suffix + ".tmp") - try: - with tmp.open("w") as fp: - print(content, file=fp, end="") - tmp.rename(filename) - except Exception as ex: - tmp.unlink() - raise + tmp = filename.with_suffix(filename.suffix + ".tmp") + try: + with tmp.open("w") as fp: + print(content, file=fp, end="") + tmp.rename(filename) + except Exception as ex: + tmp.unlink() + raise + def generate(filename, cmd, trailer): - print("Generate %s" % filename) - lcitool=subprocess.run(cmd, capture_output=True) + print("Generate %s" % filename) + lcitool = subprocess.run(cmd, capture_output=True) - if lcitool.returncode != 0: - raise Exception("Failed to generate %s: %s" % (filename, lcitool.stderr)) + if lcitool.returncode != 0: + raise Exception("Failed to generate %s: %s" % (filename, lcitool.stderr)) + + content = lcitool.stdout.decode("utf8") + if trailer is not None: + content += trailer + atomic_write(filename, content) - content = lcitool.stdout.decode("utf8") - if trailer is not None: - content += trailer - atomic_write(filename, content) def generate_dockerfile(host, target, cross=None, trailer=None): - filename = Path(src_dir, "tests", "docker", "dockerfiles", host + ".docker") - cmd = lcitool_cmd + ["dockerfile"] - if cross is not None: - cmd.extend(["--cross", cross]) - cmd.extend([target, "qemu"]) - generate(filename, cmd, trailer) + filename = Path(src_dir, "tests", "docker", "dockerfiles", host + ".docker") + cmd = lcitool_cmd + ["dockerfile"] + if cross is not None: + cmd.extend(["--cross", cross]) + cmd.extend([target, "qemu"]) + generate(filename, cmd, trailer) + def generate_cirrus(target, trailer=None): - filename = Path(src_dir, ".gitlab-ci.d", "cirrus", target + ".vars") - cmd = lcitool_cmd + ["variables", target, "qemu"] - generate(filename, cmd, trailer) + filename = Path(src_dir, ".gitlab-ci.d", "cirrus", target + ".vars") + cmd = lcitool_cmd + ["variables", target, "qemu"] + generate(filename, cmd, trailer) + ubuntu2004_tsanhack = [ - "# Apply patch https://reviews.llvm.org/D75820\n", - "# This is required for TSan in clang-10 to compile with QEMU.\n", - "RUN sed -i 's/^const/static const/g' /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h\n" + "# Apply patch https://reviews.llvm.org/D75820\n", + "# This is required for TSan in clang-10 to compile with QEMU.\n", + "RUN sed -i 's/^const/static const/g' /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h\n" ] -def debian_cross_build(prefix, targets): - conf = "ENV QEMU_CONFIGURE_OPTS --cross-prefix=%s\n" % (prefix) - targets = "ENV DEF_TARGET_LIST %s\n" % (targets) - return "".join([conf, targets]) +def debian_cross_build(prefix, targets): + conf = "ENV QEMU_CONFIGURE_OPTS --cross-prefix=%s\n" % (prefix) + targets = "ENV DEF_TARGET_LIST %s\n" % (targets) + return "".join([conf, targets]) + +# +# Update all the various build configurations. +# Please keep each group sorted alphabetically for easy reading. +# try: - generate_dockerfile("centos8", "centos-stream-8") - generate_dockerfile("fedora", "fedora-35") - generate_dockerfile("ubuntu2004", "ubuntu-2004", - trailer="".join(ubuntu2004_tsanhack)) - generate_dockerfile("opensuse-leap", "opensuse-leap-152") - generate_dockerfile("alpine", "alpine-edge") + # + # Standard native builds + # + generate_dockerfile("alpine", "alpine-edge") + generate_dockerfile("centos8", "centos-stream-8") + generate_dockerfile("fedora", "fedora-35") + generate_dockerfile("opensuse-leap", "opensuse-leap-152") + generate_dockerfile("ubuntu2004", "ubuntu-2004", + trailer="".join(ubuntu2004_tsanhack)) - generate_dockerfile("debian-arm64-cross", "debian-11", - cross="aarch64", - trailer=debian_cross_build("aarch64-linux-gnu-", - "aarch64-softmmu,aarch64-linux-user")) + # + # Cross compiling builds + # + generate_dockerfile("debian-arm64-cross", "debian-11", + cross="aarch64", + trailer=debian_cross_build("aarch64-linux-gnu-", + "aarch64-softmmu,aarch64-linux-user")) - generate_dockerfile("debian-s390x-cross", "debian-11", - cross="s390x", - trailer=debian_cross_build("s390x-linux-gnu-", - "s390x-softmmu,s390x-linux-user")) + generate_dockerfile("debian-s390x-cross", "debian-11", + cross="s390x", + trailer=debian_cross_build("s390x-linux-gnu-", + "s390x-softmmu,s390x-linux-user")) - generate_cirrus("freebsd-12") - generate_cirrus("freebsd-13") - generate_cirrus("macos-11") + # + # Cirrus packages lists for GitLab + # + generate_cirrus("freebsd-12") + generate_cirrus("freebsd-13") + generate_cirrus("macos-11") - sys.exit(0) + sys.exit(0) except Exception as ex: - print(str(ex), file=sys.stderr) - sys.exit(1) + print(str(ex), file=sys.stderr) + sys.exit(1) From 18ad049d90feede8affee797822fca41d1472b71 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alex=20Benn=C3=A9e?= Date: Fri, 27 May 2022 16:35:36 +0100 Subject: [PATCH 06/33] tests/docker: update debian-armhf-cross with lcitool MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Use lcitool to update debian-armhf-cross to a Debian 11 based system. Signed-off-by: Alex Bennée Reviewed-by: Daniel P. Berrangé Message-Id: <20220527153603.887929-7-alex.bennee@linaro.org> --- .gitlab-ci.d/container-cross.yml | 3 +- tests/docker/Makefile.include | 1 - .../dockerfiles/debian-armhf-cross.docker | 184 +++++++++++++++--- tests/lcitool/refresh | 5 + 4 files changed, 166 insertions(+), 27 deletions(-) diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml index ac15fce9b6..4d1830f3fc 100644 --- a/.gitlab-ci.d/container-cross.yml +++ b/.gitlab-ci.d/container-cross.yml @@ -34,8 +34,7 @@ armel-debian-cross-container: armhf-debian-cross-container: extends: .container_job_template - stage: containers-layer2 - needs: ['amd64-debian10-container'] + stage: containers variables: NAME: debian-armhf-cross diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include index ca2157db46..d6e0710554 100644 --- a/tests/docker/Makefile.include +++ b/tests/docker/Makefile.include @@ -90,7 +90,6 @@ endif docker-image-debian-alpha-cross: docker-image-debian10 docker-image-debian-armel-cross: docker-image-debian10 -docker-image-debian-armhf-cross: docker-image-debian10 docker-image-debian-hppa-cross: docker-image-debian10 docker-image-debian-m68k-cross: docker-image-debian10 docker-image-debian-mips-cross: docker-image-debian10 diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker index 25d7618833..a2ebce96f8 100644 --- a/tests/docker/dockerfiles/debian-armhf-cross.docker +++ b/tests/docker/dockerfiles/debian-armhf-cross.docker @@ -1,29 +1,165 @@ +# THIS FILE WAS AUTO-GENERATED # -# Docker armhf cross-compiler target +# $ lcitool dockerfile --layers all --cross armv7l debian-11 qemu # -# This docker target builds on the debian Stretch base image. -# -FROM qemu/debian10 +# https://gitlab.com/libvirt/libvirt-ci -# Add the foreign architecture we want and install dependencies -RUN dpkg --add-architecture armhf -RUN apt update && \ - DEBIAN_FRONTEND=noninteractive eatmydata \ - apt install -y --no-install-recommends \ - crossbuild-essential-armhf -RUN apt update && \ - DEBIAN_FRONTEND=noninteractive eatmydata \ - apt build-dep -yy -a armhf --arch-only qemu +FROM docker.io/library/debian:11-slim -# Specify the cross prefix for this image (see tests/docker/common.rc) +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get install -y eatmydata && \ + eatmydata apt-get dist-upgrade -y && \ + eatmydata apt-get install --no-install-recommends -y \ + bash \ + bc \ + bsdextrautils \ + bzip2 \ + ca-certificates \ + ccache \ + dbus \ + debianutils \ + diffutils \ + exuberant-ctags \ + findutils \ + gcovr \ + genisoimage \ + gettext \ + git \ + hostname \ + libpcre2-dev \ + libspice-protocol-dev \ + llvm \ + locales \ + make \ + meson \ + ncat \ + ninja-build \ + openssh-client \ + perl-base \ + pkgconf \ + python3 \ + python3-numpy \ + python3-opencv \ + python3-pillow \ + python3-pip \ + python3-sphinx \ + python3-sphinx-rtd-theme \ + python3-venv \ + python3-yaml \ + rpm2cpio \ + sed \ + sparse \ + tar \ + tesseract-ocr \ + tesseract-ocr-eng \ + texinfo && \ + eatmydata apt-get autoremove -y && \ + eatmydata apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales + +ENV LANG "en_US.UTF-8" +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" + +RUN export DEBIAN_FRONTEND=noninteractive && \ + dpkg --add-architecture armhf && \ + eatmydata apt-get update && \ + eatmydata apt-get dist-upgrade -y && \ + eatmydata apt-get install --no-install-recommends -y dpkg-dev && \ + eatmydata apt-get install --no-install-recommends -y \ + g++-arm-linux-gnueabihf \ + gcc-arm-linux-gnueabihf \ + libaio-dev:armhf \ + libasan5:armhf \ + libasound2-dev:armhf \ + libattr1-dev:armhf \ + libbpf-dev:armhf \ + libbrlapi-dev:armhf \ + libbz2-dev:armhf \ + libc6-dev:armhf \ + libcacard-dev:armhf \ + libcap-ng-dev:armhf \ + libcapstone-dev:armhf \ + libcurl4-gnutls-dev:armhf \ + libdaxctl-dev:armhf \ + libdrm-dev:armhf \ + libepoxy-dev:armhf \ + libfdt-dev:armhf \ + libffi-dev:armhf \ + libfuse3-dev:armhf \ + libgbm-dev:armhf \ + libgcrypt20-dev:armhf \ + libglib2.0-dev:armhf \ + libglusterfs-dev:armhf \ + libgnutls28-dev:armhf \ + libgtk-3-dev:armhf \ + libibumad-dev:armhf \ + libibverbs-dev:armhf \ + libiscsi-dev:armhf \ + libjemalloc-dev:armhf \ + libjpeg62-turbo-dev:armhf \ + liblttng-ust-dev:armhf \ + liblzo2-dev:armhf \ + libncursesw5-dev:armhf \ + libnfs-dev:armhf \ + libnuma-dev:armhf \ + libpam0g-dev:armhf \ + libpixman-1-dev:armhf \ + libpng-dev:armhf \ + libpulse-dev:armhf \ + librbd-dev:armhf \ + librdmacm-dev:armhf \ + libsasl2-dev:armhf \ + libsdl2-dev:armhf \ + libsdl2-image-dev:armhf \ + libseccomp-dev:armhf \ + libselinux1-dev:armhf \ + libslirp-dev:armhf \ + libsnappy-dev:armhf \ + libspice-server-dev:armhf \ + libssh-gcrypt-dev:armhf \ + libsystemd-dev:armhf \ + libtasn1-6-dev:armhf \ + libubsan1:armhf \ + libudev-dev:armhf \ + liburing-dev:armhf \ + libusb-1.0-0-dev:armhf \ + libusbredirhost-dev:armhf \ + libvdeplug-dev:armhf \ + libvirglrenderer-dev:armhf \ + libvte-2.91-dev:armhf \ + libxen-dev:armhf \ + libzstd-dev:armhf \ + nettle-dev:armhf \ + systemtap-sdt-dev:armhf \ + xfslibs-dev:armhf \ + zlib1g-dev:armhf && \ + eatmydata apt-get autoremove -y && \ + eatmydata apt-get autoclean -y && \ + mkdir -p /usr/local/share/meson/cross && \ + echo "[binaries]\n\ +c = '/usr/bin/arm-linux-gnueabihf-gcc'\n\ +ar = '/usr/bin/arm-linux-gnueabihf-gcc-ar'\n\ +strip = '/usr/bin/arm-linux-gnueabihf-strip'\n\ +pkgconfig = '/usr/bin/arm-linux-gnueabihf-pkg-config'\n\ +\n\ +[host_machine]\n\ +system = 'linux'\n\ +cpu_family = 'arm'\n\ +cpu = 'armhf'\n\ +endian = 'little'" > /usr/local/share/meson/cross/arm-linux-gnueabihf && \ + dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/arm-linux-gnueabihf-c++ && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/arm-linux-gnueabihf-cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/arm-linux-gnueabihf-g++ && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/arm-linux-gnueabihf-gcc + +ENV ABI "arm-linux-gnueabihf" +ENV MESON_OPTS "--cross-file=arm-linux-gnueabihf" ENV QEMU_CONFIGURE_OPTS --cross-prefix=arm-linux-gnueabihf- -ENV DEF_TARGET_LIST arm-softmmu,arm-linux-user,armeb-linux-user - -RUN apt update && \ - DEBIAN_FRONTEND=noninteractive eatmydata \ - apt install -y --no-install-recommends \ - libbz2-dev:armhf \ - liblzo2-dev:armhf \ - librdmacm-dev:armhf \ - libsnappy-dev:armhf \ - libxen-dev:armhf +ENV DEF_TARGET_LIST arm-softmmu,arm-linux-user diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh index dc1fc21ef9..7e4a92f630 100755 --- a/tests/lcitool/refresh +++ b/tests/lcitool/refresh @@ -105,6 +105,11 @@ try: trailer=debian_cross_build("aarch64-linux-gnu-", "aarch64-softmmu,aarch64-linux-user")) + generate_dockerfile("debian-armhf-cross", "debian-11", + cross="armv7l", + trailer=debian_cross_build("arm-linux-gnueabihf-", + "arm-softmmu,arm-linux-user")) + generate_dockerfile("debian-s390x-cross", "debian-11", cross="s390x", trailer=debian_cross_build("s390x-linux-gnu-", From 94806241878e2f8908a04f82fe4086ce56690d80 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alex=20Benn=C3=A9e?= Date: Fri, 27 May 2022 16:35:37 +0100 Subject: [PATCH 07/33] tests/docker: update debian-armel-cross with lcitool MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Use lcitool to update debian-armel-cross to a Debian 11 based system. Signed-off-by: Alex Bennée Reviewed-by: Daniel P. Berrangé Message-Id: <20220527153603.887929-8-alex.bennee@linaro.org> --- .gitlab-ci.d/container-cross.yml | 3 +- tests/docker/Makefile.include | 1 - .../dockerfiles/debian-armel-cross.docker | 178 ++++++++++++++++-- tests/lcitool/refresh | 5 + 4 files changed, 164 insertions(+), 23 deletions(-) diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml index 4d1830f3fc..caef7decf4 100644 --- a/.gitlab-ci.d/container-cross.yml +++ b/.gitlab-ci.d/container-cross.yml @@ -27,8 +27,7 @@ arm64-debian-cross-container: armel-debian-cross-container: extends: .container_job_template - stage: containers-layer2 - needs: ['amd64-debian10-container'] + stage: containers variables: NAME: debian-armel-cross diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include index d6e0710554..d9109bcc77 100644 --- a/tests/docker/Makefile.include +++ b/tests/docker/Makefile.include @@ -89,7 +89,6 @@ DOCKER_PARTIAL_IMAGES += fedora endif docker-image-debian-alpha-cross: docker-image-debian10 -docker-image-debian-armel-cross: docker-image-debian10 docker-image-debian-hppa-cross: docker-image-debian10 docker-image-debian-m68k-cross: docker-image-debian10 docker-image-debian-mips-cross: docker-image-debian10 diff --git a/tests/docker/dockerfiles/debian-armel-cross.docker b/tests/docker/dockerfiles/debian-armel-cross.docker index b7b1a3585f..a6153e5a83 100644 --- a/tests/docker/dockerfiles/debian-armel-cross.docker +++ b/tests/docker/dockerfiles/debian-armel-cross.docker @@ -1,26 +1,164 @@ +# THIS FILE WAS AUTO-GENERATED # -# Docker armel cross-compiler target +# $ lcitool dockerfile --layers all --cross armv6l debian-11 qemu # -# This docker target builds on the debian Stretch base image. -# -FROM qemu/debian10 -MAINTAINER Philippe Mathieu-Daudé +# https://gitlab.com/libvirt/libvirt-ci -# Add the foreign architecture we want and install dependencies -RUN dpkg --add-architecture armel && \ - apt update && \ - apt install -yy crossbuild-essential-armel && \ - DEBIAN_FRONTEND=noninteractive eatmydata \ - apt build-dep -yy -a armel --arch-only qemu +FROM docker.io/library/debian:11-slim -# Specify the cross prefix for this image (see tests/docker/common.rc) +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get install -y eatmydata && \ + eatmydata apt-get dist-upgrade -y && \ + eatmydata apt-get install --no-install-recommends -y \ + bash \ + bc \ + bsdextrautils \ + bzip2 \ + ca-certificates \ + ccache \ + dbus \ + debianutils \ + diffutils \ + exuberant-ctags \ + findutils \ + gcovr \ + genisoimage \ + gettext \ + git \ + hostname \ + libpcre2-dev \ + libspice-protocol-dev \ + llvm \ + locales \ + make \ + meson \ + ncat \ + ninja-build \ + openssh-client \ + perl-base \ + pkgconf \ + python3 \ + python3-numpy \ + python3-opencv \ + python3-pillow \ + python3-pip \ + python3-sphinx \ + python3-sphinx-rtd-theme \ + python3-venv \ + python3-yaml \ + rpm2cpio \ + sed \ + sparse \ + tar \ + tesseract-ocr \ + tesseract-ocr-eng \ + texinfo && \ + eatmydata apt-get autoremove -y && \ + eatmydata apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales + +ENV LANG "en_US.UTF-8" +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" + +RUN export DEBIAN_FRONTEND=noninteractive && \ + dpkg --add-architecture armel && \ + eatmydata apt-get update && \ + eatmydata apt-get dist-upgrade -y && \ + eatmydata apt-get install --no-install-recommends -y dpkg-dev && \ + eatmydata apt-get install --no-install-recommends -y \ + g++-arm-linux-gnueabi \ + gcc-arm-linux-gnueabi \ + libaio-dev:armel \ + libasan5:armel \ + libasound2-dev:armel \ + libattr1-dev:armel \ + libbpf-dev:armel \ + libbrlapi-dev:armel \ + libbz2-dev:armel \ + libc6-dev:armel \ + libcacard-dev:armel \ + libcap-ng-dev:armel \ + libcapstone-dev:armel \ + libcurl4-gnutls-dev:armel \ + libdaxctl-dev:armel \ + libdrm-dev:armel \ + libepoxy-dev:armel \ + libfdt-dev:armel \ + libffi-dev:armel \ + libfuse3-dev:armel \ + libgbm-dev:armel \ + libgcrypt20-dev:armel \ + libglib2.0-dev:armel \ + libglusterfs-dev:armel \ + libgnutls28-dev:armel \ + libgtk-3-dev:armel \ + libibumad-dev:armel \ + libibverbs-dev:armel \ + libiscsi-dev:armel \ + libjemalloc-dev:armel \ + libjpeg62-turbo-dev:armel \ + liblttng-ust-dev:armel \ + liblzo2-dev:armel \ + libncursesw5-dev:armel \ + libnfs-dev:armel \ + libnuma-dev:armel \ + libpam0g-dev:armel \ + libpixman-1-dev:armel \ + libpng-dev:armel \ + libpulse-dev:armel \ + librbd-dev:armel \ + librdmacm-dev:armel \ + libsasl2-dev:armel \ + libsdl2-dev:armel \ + libsdl2-image-dev:armel \ + libseccomp-dev:armel \ + libselinux1-dev:armel \ + libslirp-dev:armel \ + libsnappy-dev:armel \ + libspice-server-dev:armel \ + libssh-gcrypt-dev:armel \ + libsystemd-dev:armel \ + libtasn1-6-dev:armel \ + libubsan1:armel \ + libudev-dev:armel \ + liburing-dev:armel \ + libusb-1.0-0-dev:armel \ + libusbredirhost-dev:armel \ + libvdeplug-dev:armel \ + libvirglrenderer-dev:armel \ + libvte-2.91-dev:armel \ + libzstd-dev:armel \ + nettle-dev:armel \ + systemtap-sdt-dev:armel \ + xfslibs-dev:armel \ + zlib1g-dev:armel && \ + eatmydata apt-get autoremove -y && \ + eatmydata apt-get autoclean -y && \ + mkdir -p /usr/local/share/meson/cross && \ + echo "[binaries]\n\ +c = '/usr/bin/arm-linux-gnueabi-gcc'\n\ +ar = '/usr/bin/arm-linux-gnueabi-gcc-ar'\n\ +strip = '/usr/bin/arm-linux-gnueabi-strip'\n\ +pkgconfig = '/usr/bin/arm-linux-gnueabi-pkg-config'\n\ +\n\ +[host_machine]\n\ +system = 'linux'\n\ +cpu_family = 'arm'\n\ +cpu = 'arm'\n\ +endian = 'little'" > /usr/local/share/meson/cross/arm-linux-gnueabi && \ + dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/arm-linux-gnueabi-c++ && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/arm-linux-gnueabi-cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/arm-linux-gnueabi-g++ && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/arm-linux-gnueabi-gcc + +ENV ABI "arm-linux-gnueabi" +ENV MESON_OPTS "--cross-file=arm-linux-gnueabi" ENV QEMU_CONFIGURE_OPTS --cross-prefix=arm-linux-gnueabi- ENV DEF_TARGET_LIST arm-softmmu,arm-linux-user,armeb-linux-user - -RUN apt update && \ - DEBIAN_FRONTEND=noninteractive eatmydata \ - apt install -y --no-install-recommends \ - libbz2-dev:armel \ - liblzo2-dev:armel \ - librdmacm-dev:armel \ - libsnappy-dev:armel diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh index 7e4a92f630..4dc5527234 100755 --- a/tests/lcitool/refresh +++ b/tests/lcitool/refresh @@ -105,6 +105,11 @@ try: trailer=debian_cross_build("aarch64-linux-gnu-", "aarch64-softmmu,aarch64-linux-user")) + generate_dockerfile("debian-armel-cross", "debian-11", + cross="armv6l", + trailer=debian_cross_build("arm-linux-gnueabi-", + "arm-softmmu,arm-linux-user,armeb-linux-user")) + generate_dockerfile("debian-armhf-cross", "debian-11", cross="armv7l", trailer=debian_cross_build("arm-linux-gnueabihf-", From 432ae739c6ce6f31f74da62be34325890ded278b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alex=20Benn=C3=A9e?= Date: Fri, 27 May 2022 16:35:38 +0100 Subject: [PATCH 08/33] tests/docker: update debian-mipsel-cross with lcitool MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Use lcitool to update debian-mipsel-cross to a Debian 11 based system. Signed-off-by: Alex Bennée Reviewed-by: Daniel P. Berrangé Message-Id: <20220527153603.887929-9-alex.bennee@linaro.org> --- .gitlab-ci.d/container-cross.yml | 3 +- tests/docker/Makefile.include | 1 - .../dockerfiles/debian-mipsel-cross.docker | 179 +++++++++++++++--- tests/lcitool/refresh | 5 + 4 files changed, 161 insertions(+), 27 deletions(-) diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml index caef7decf4..1a533e6fc0 100644 --- a/.gitlab-ci.d/container-cross.yml +++ b/.gitlab-ci.d/container-cross.yml @@ -102,8 +102,7 @@ mips-debian-cross-container: mipsel-debian-cross-container: extends: .container_job_template - stage: containers-layer2 - needs: ['amd64-debian10-container'] + stage: containers variables: NAME: debian-mipsel-cross diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include index d9109bcc77..0ac5975419 100644 --- a/tests/docker/Makefile.include +++ b/tests/docker/Makefile.include @@ -94,7 +94,6 @@ docker-image-debian-m68k-cross: docker-image-debian10 docker-image-debian-mips-cross: docker-image-debian10 docker-image-debian-mips64-cross: docker-image-debian10 docker-image-debian-mips64el-cross: docker-image-debian10 -docker-image-debian-mipsel-cross: docker-image-debian10 docker-image-debian-ppc64el-cross: docker-image-debian10 docker-image-debian-sh4-cross: docker-image-debian10 docker-image-debian-sparc64-cross: docker-image-debian10 diff --git a/tests/docker/dockerfiles/debian-mipsel-cross.docker b/tests/docker/dockerfiles/debian-mipsel-cross.docker index 0e5dd42d3c..b6d99ae324 100644 --- a/tests/docker/dockerfiles/debian-mipsel-cross.docker +++ b/tests/docker/dockerfiles/debian-mipsel-cross.docker @@ -1,31 +1,162 @@ +# THIS FILE WAS AUTO-GENERATED # -# Docker mipsel cross-compiler target +# $ lcitool dockerfile --layers all --cross mipsel debian-11 qemu # -# This docker target builds on the debian Stretch base image. -# -FROM qemu/debian10 +# https://gitlab.com/libvirt/libvirt-ci -MAINTAINER Philippe Mathieu-Daudé +FROM docker.io/library/debian:11-slim -# Add the foreign architecture we want and install dependencies -RUN dpkg --add-architecture mipsel -RUN apt update && \ - DEBIAN_FRONTEND=noninteractive eatmydata \ - apt install -y --no-install-recommends \ - gcc-mipsel-linux-gnu +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get install -y eatmydata && \ + eatmydata apt-get dist-upgrade -y && \ + eatmydata apt-get install --no-install-recommends -y \ + bash \ + bc \ + bsdextrautils \ + bzip2 \ + ca-certificates \ + ccache \ + dbus \ + debianutils \ + diffutils \ + exuberant-ctags \ + findutils \ + gcovr \ + genisoimage \ + gettext \ + git \ + hostname \ + libpcre2-dev \ + libspice-protocol-dev \ + llvm \ + locales \ + make \ + meson \ + ncat \ + ninja-build \ + openssh-client \ + perl-base \ + pkgconf \ + python3 \ + python3-numpy \ + python3-opencv \ + python3-pillow \ + python3-pip \ + python3-sphinx \ + python3-sphinx-rtd-theme \ + python3-venv \ + python3-yaml \ + rpm2cpio \ + sed \ + sparse \ + tar \ + tesseract-ocr \ + tesseract-ocr-eng \ + texinfo && \ + eatmydata apt-get autoremove -y && \ + eatmydata apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales -RUN apt update && \ - DEBIAN_FRONTEND=noninteractive eatmydata \ - apt build-dep -yy -a mipsel --arch-only qemu +ENV LANG "en_US.UTF-8" +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" -# Specify the cross prefix for this image (see tests/docker/common.rc) +RUN export DEBIAN_FRONTEND=noninteractive && \ + dpkg --add-architecture mipsel && \ + eatmydata apt-get update && \ + eatmydata apt-get dist-upgrade -y && \ + eatmydata apt-get install --no-install-recommends -y dpkg-dev && \ + eatmydata apt-get install --no-install-recommends -y \ + g++-mipsel-linux-gnu \ + gcc-mipsel-linux-gnu \ + libaio-dev:mipsel \ + libasound2-dev:mipsel \ + libattr1-dev:mipsel \ + libbpf-dev:mipsel \ + libbrlapi-dev:mipsel \ + libbz2-dev:mipsel \ + libc6-dev:mipsel \ + libcacard-dev:mipsel \ + libcap-ng-dev:mipsel \ + libcapstone-dev:mipsel \ + libcurl4-gnutls-dev:mipsel \ + libdaxctl-dev:mipsel \ + libdrm-dev:mipsel \ + libepoxy-dev:mipsel \ + libfdt-dev:mipsel \ + libffi-dev:mipsel \ + libfuse3-dev:mipsel \ + libgbm-dev:mipsel \ + libgcrypt20-dev:mipsel \ + libglib2.0-dev:mipsel \ + libglusterfs-dev:mipsel \ + libgnutls28-dev:mipsel \ + libgtk-3-dev:mipsel \ + libibumad-dev:mipsel \ + libibverbs-dev:mipsel \ + libiscsi-dev:mipsel \ + libjemalloc-dev:mipsel \ + libjpeg62-turbo-dev:mipsel \ + liblttng-ust-dev:mipsel \ + liblzo2-dev:mipsel \ + libncursesw5-dev:mipsel \ + libnfs-dev:mipsel \ + libnuma-dev:mipsel \ + libpam0g-dev:mipsel \ + libpixman-1-dev:mipsel \ + libpng-dev:mipsel \ + libpulse-dev:mipsel \ + librbd-dev:mipsel \ + librdmacm-dev:mipsel \ + libsasl2-dev:mipsel \ + libsdl2-dev:mipsel \ + libsdl2-image-dev:mipsel \ + libseccomp-dev:mipsel \ + libselinux1-dev:mipsel \ + libslirp-dev:mipsel \ + libsnappy-dev:mipsel \ + libspice-server-dev:mipsel \ + libssh-gcrypt-dev:mipsel \ + libsystemd-dev:mipsel \ + libtasn1-6-dev:mipsel \ + libudev-dev:mipsel \ + liburing-dev:mipsel \ + libusb-1.0-0-dev:mipsel \ + libusbredirhost-dev:mipsel \ + libvdeplug-dev:mipsel \ + libvirglrenderer-dev:mipsel \ + libvte-2.91-dev:mipsel \ + libzstd-dev:mipsel \ + nettle-dev:mipsel \ + systemtap-sdt-dev:mipsel \ + xfslibs-dev:mipsel \ + zlib1g-dev:mipsel && \ + eatmydata apt-get autoremove -y && \ + eatmydata apt-get autoclean -y && \ + mkdir -p /usr/local/share/meson/cross && \ + echo "[binaries]\n\ +c = '/usr/bin/mipsel-linux-gnu-gcc'\n\ +ar = '/usr/bin/mipsel-linux-gnu-gcc-ar'\n\ +strip = '/usr/bin/mipsel-linux-gnu-strip'\n\ +pkgconfig = '/usr/bin/mipsel-linux-gnu-pkg-config'\n\ +\n\ +[host_machine]\n\ +system = 'linux'\n\ +cpu_family = 'mips'\n\ +cpu = 'mipsel'\n\ +endian = 'little'" > /usr/local/share/meson/cross/mipsel-linux-gnu && \ + dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/mipsel-linux-gnu-c++ && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/mipsel-linux-gnu-cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/mipsel-linux-gnu-g++ && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/mipsel-linux-gnu-gcc + +ENV ABI "mipsel-linux-gnu" +ENV MESON_OPTS "--cross-file=mipsel-linux-gnu" ENV QEMU_CONFIGURE_OPTS --cross-prefix=mipsel-linux-gnu- - -# Install extra libraries to increase code coverage -RUN apt update && \ - DEBIAN_FRONTEND=noninteractive eatmydata \ - apt install -y --no-install-recommends \ - libbz2-dev:mipsel \ - liblzo2-dev:mipsel \ - librdmacm-dev:mipsel \ - libsnappy-dev:mipsel +ENV DEF_TARGET_LIST mipsel-softmmu,mipsel-linux-user diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh index 4dc5527234..a4b7452189 100755 --- a/tests/lcitool/refresh +++ b/tests/lcitool/refresh @@ -115,6 +115,11 @@ try: trailer=debian_cross_build("arm-linux-gnueabihf-", "arm-softmmu,arm-linux-user")) + generate_dockerfile("debian-mipsel-cross", "debian-11", + cross="mipsel", + trailer=debian_cross_build("mipsel-linux-gnu-", + "mipsel-softmmu,mipsel-linux-user")) + generate_dockerfile("debian-s390x-cross", "debian-11", cross="s390x", trailer=debian_cross_build("s390x-linux-gnu-", From 1e834d1714e72579d595d5395a3bb35b12a05d09 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alex=20Benn=C3=A9e?= Date: Fri, 27 May 2022 16:35:39 +0100 Subject: [PATCH 09/33] tests/docker: update debian-mips64el-cross with lcitool MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Use lcitool to update debian-mips64el-cross to a Debian 11 based system. Signed-off-by: Alex Bennée Reviewed-by: Daniel P. Berrangé Message-Id: <20220527153603.887929-10-alex.bennee@linaro.org> --- .gitlab-ci.d/container-cross.yml | 3 +- tests/docker/Makefile.include | 1 - .../dockerfiles/debian-mips64el-cross.docker | 177 +++++++++++++++--- tests/lcitool/refresh | 5 + 4 files changed, 159 insertions(+), 27 deletions(-) diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml index 1a533e6fc0..411dc06bf8 100644 --- a/.gitlab-ci.d/container-cross.yml +++ b/.gitlab-ci.d/container-cross.yml @@ -88,8 +88,7 @@ mips64-debian-cross-container: mips64el-debian-cross-container: extends: .container_job_template - stage: containers-layer2 - needs: ['amd64-debian10-container'] + stage: containers variables: NAME: debian-mips64el-cross diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include index 0ac5975419..d9f37ae8fa 100644 --- a/tests/docker/Makefile.include +++ b/tests/docker/Makefile.include @@ -93,7 +93,6 @@ docker-image-debian-hppa-cross: docker-image-debian10 docker-image-debian-m68k-cross: docker-image-debian10 docker-image-debian-mips-cross: docker-image-debian10 docker-image-debian-mips64-cross: docker-image-debian10 -docker-image-debian-mips64el-cross: docker-image-debian10 docker-image-debian-ppc64el-cross: docker-image-debian10 docker-image-debian-sh4-cross: docker-image-debian10 docker-image-debian-sparc64-cross: docker-image-debian10 diff --git a/tests/docker/dockerfiles/debian-mips64el-cross.docker b/tests/docker/dockerfiles/debian-mips64el-cross.docker index c990b683b7..b02dcb7fd9 100644 --- a/tests/docker/dockerfiles/debian-mips64el-cross.docker +++ b/tests/docker/dockerfiles/debian-mips64el-cross.docker @@ -1,33 +1,162 @@ +# THIS FILE WAS AUTO-GENERATED # -# Docker mips64el cross-compiler target -# -# This docker target builds on the debian Stretch base image. +# $ lcitool dockerfile --layers all --cross mips64el debian-11 qemu # +# https://gitlab.com/libvirt/libvirt-ci -FROM qemu/debian10 +FROM docker.io/library/debian:11-slim -MAINTAINER Philippe Mathieu-Daudé +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get install -y eatmydata && \ + eatmydata apt-get dist-upgrade -y && \ + eatmydata apt-get install --no-install-recommends -y \ + bash \ + bc \ + bsdextrautils \ + bzip2 \ + ca-certificates \ + ccache \ + dbus \ + debianutils \ + diffutils \ + exuberant-ctags \ + findutils \ + gcovr \ + genisoimage \ + gettext \ + git \ + hostname \ + libpcre2-dev \ + libspice-protocol-dev \ + llvm \ + locales \ + make \ + meson \ + ncat \ + ninja-build \ + openssh-client \ + perl-base \ + pkgconf \ + python3 \ + python3-numpy \ + python3-opencv \ + python3-pillow \ + python3-pip \ + python3-sphinx \ + python3-sphinx-rtd-theme \ + python3-venv \ + python3-yaml \ + rpm2cpio \ + sed \ + sparse \ + tar \ + tesseract-ocr \ + tesseract-ocr-eng \ + texinfo && \ + eatmydata apt-get autoremove -y && \ + eatmydata apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales -# Add the foreign architecture we want and install dependencies -RUN dpkg --add-architecture mips64el && \ - apt update && \ - DEBIAN_FRONTEND=noninteractive eatmydata \ - apt install -y --no-install-recommends \ - gcc-mips64el-linux-gnuabi64 +ENV LANG "en_US.UTF-8" +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" -RUN apt update && \ - DEBIAN_FRONTEND=noninteractive eatmydata \ - apt build-dep -yy -a mips64el --arch-only qemu +RUN export DEBIAN_FRONTEND=noninteractive && \ + dpkg --add-architecture mips64el && \ + eatmydata apt-get update && \ + eatmydata apt-get dist-upgrade -y && \ + eatmydata apt-get install --no-install-recommends -y dpkg-dev && \ + eatmydata apt-get install --no-install-recommends -y \ + g++-mips64el-linux-gnuabi64 \ + gcc-mips64el-linux-gnuabi64 \ + libaio-dev:mips64el \ + libasound2-dev:mips64el \ + libattr1-dev:mips64el \ + libbpf-dev:mips64el \ + libbrlapi-dev:mips64el \ + libbz2-dev:mips64el \ + libc6-dev:mips64el \ + libcacard-dev:mips64el \ + libcap-ng-dev:mips64el \ + libcapstone-dev:mips64el \ + libcurl4-gnutls-dev:mips64el \ + libdaxctl-dev:mips64el \ + libdrm-dev:mips64el \ + libepoxy-dev:mips64el \ + libfdt-dev:mips64el \ + libffi-dev:mips64el \ + libfuse3-dev:mips64el \ + libgbm-dev:mips64el \ + libgcrypt20-dev:mips64el \ + libglib2.0-dev:mips64el \ + libglusterfs-dev:mips64el \ + libgnutls28-dev:mips64el \ + libgtk-3-dev:mips64el \ + libibumad-dev:mips64el \ + libibverbs-dev:mips64el \ + libiscsi-dev:mips64el \ + libjemalloc-dev:mips64el \ + libjpeg62-turbo-dev:mips64el \ + liblttng-ust-dev:mips64el \ + liblzo2-dev:mips64el \ + libncursesw5-dev:mips64el \ + libnfs-dev:mips64el \ + libnuma-dev:mips64el \ + libpam0g-dev:mips64el \ + libpixman-1-dev:mips64el \ + libpng-dev:mips64el \ + libpulse-dev:mips64el \ + librbd-dev:mips64el \ + librdmacm-dev:mips64el \ + libsasl2-dev:mips64el \ + libsdl2-dev:mips64el \ + libsdl2-image-dev:mips64el \ + libseccomp-dev:mips64el \ + libselinux1-dev:mips64el \ + libslirp-dev:mips64el \ + libsnappy-dev:mips64el \ + libspice-server-dev:mips64el \ + libssh-gcrypt-dev:mips64el \ + libsystemd-dev:mips64el \ + libtasn1-6-dev:mips64el \ + libudev-dev:mips64el \ + liburing-dev:mips64el \ + libusb-1.0-0-dev:mips64el \ + libusbredirhost-dev:mips64el \ + libvdeplug-dev:mips64el \ + libvirglrenderer-dev:mips64el \ + libvte-2.91-dev:mips64el \ + libzstd-dev:mips64el \ + nettle-dev:mips64el \ + systemtap-sdt-dev:mips64el \ + xfslibs-dev:mips64el \ + zlib1g-dev:mips64el && \ + eatmydata apt-get autoremove -y && \ + eatmydata apt-get autoclean -y && \ + mkdir -p /usr/local/share/meson/cross && \ + echo "[binaries]\n\ +c = '/usr/bin/mips64el-linux-gnuabi64-gcc'\n\ +ar = '/usr/bin/mips64el-linux-gnuabi64-gcc-ar'\n\ +strip = '/usr/bin/mips64el-linux-gnuabi64-strip'\n\ +pkgconfig = '/usr/bin/mips64el-linux-gnuabi64-pkg-config'\n\ +\n\ +[host_machine]\n\ +system = 'linux'\n\ +cpu_family = 'mips64'\n\ +cpu = 'mips64el'\n\ +endian = 'little'" > /usr/local/share/meson/cross/mips64el-linux-gnuabi64 && \ + dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/mips64el-linux-gnuabi64-c++ && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/mips64el-linux-gnuabi64-cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/mips64el-linux-gnuabi64-g++ && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/mips64el-linux-gnuabi64-gcc -# Specify the cross prefix for this image (see tests/docker/common.rc) +ENV ABI "mips64el-linux-gnuabi64" +ENV MESON_OPTS "--cross-file=mips64el-linux-gnuabi64" ENV QEMU_CONFIGURE_OPTS --cross-prefix=mips64el-linux-gnuabi64- ENV DEF_TARGET_LIST mips64el-softmmu,mips64el-linux-user - -# Install extra libraries to increase code coverage -RUN apt update && \ - DEBIAN_FRONTEND=noninteractive eatmydata \ - apt install -y --no-install-recommends \ - libbz2-dev:mips64el \ - liblzo2-dev:mips64el \ - librdmacm-dev:mips64el \ - libsnappy-dev:mips64el diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh index a4b7452189..341a07e677 100755 --- a/tests/lcitool/refresh +++ b/tests/lcitool/refresh @@ -115,6 +115,11 @@ try: trailer=debian_cross_build("arm-linux-gnueabihf-", "arm-softmmu,arm-linux-user")) + generate_dockerfile("debian-mips64el-cross", "debian-11", + cross="mips64el", + trailer=debian_cross_build("mips64el-linux-gnuabi64-", + "mips64el-softmmu,mips64el-linux-user")) + generate_dockerfile("debian-mipsel-cross", "debian-11", cross="mipsel", trailer=debian_cross_build("mipsel-linux-gnu-", From 4ec740e12da8a969f1ae3f9147dde9ce3a79a506 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alex=20Benn=C3=A9e?= Date: Fri, 27 May 2022 16:35:40 +0100 Subject: [PATCH 10/33] tests/docker: update debian-ppc64el-cross with lcitool MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Use lcitool to update debian-ppc64el-cross to a Debian 11 based system. Signed-off-by: Alex Bennée Reviewed-by: Daniel P. Berrangé Message-Id: <20220527153603.887929-11-alex.bennee@linaro.org> --- .gitlab-ci.d/container-cross.yml | 3 +- tests/docker/Makefile.include | 1 - .../dockerfiles/debian-ppc64el-cross.docker | 178 +++++++++++++++--- tests/lcitool/refresh | 5 + 4 files changed, 163 insertions(+), 24 deletions(-) diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml index 411dc06bf8..147e667744 100644 --- a/.gitlab-ci.d/container-cross.yml +++ b/.gitlab-ci.d/container-cross.yml @@ -114,8 +114,7 @@ powerpc-test-cross-container: ppc64el-debian-cross-container: extends: .container_job_template - stage: containers-layer2 - needs: ['amd64-debian10-container'] + stage: containers variables: NAME: debian-ppc64el-cross diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include index d9f37ae8fa..e68f91b853 100644 --- a/tests/docker/Makefile.include +++ b/tests/docker/Makefile.include @@ -93,7 +93,6 @@ docker-image-debian-hppa-cross: docker-image-debian10 docker-image-debian-m68k-cross: docker-image-debian10 docker-image-debian-mips-cross: docker-image-debian10 docker-image-debian-mips64-cross: docker-image-debian10 -docker-image-debian-ppc64el-cross: docker-image-debian10 docker-image-debian-sh4-cross: docker-image-debian10 docker-image-debian-sparc64-cross: docker-image-debian10 diff --git a/tests/docker/dockerfiles/debian-ppc64el-cross.docker b/tests/docker/dockerfiles/debian-ppc64el-cross.docker index 5de12b01cd..bcf04bc90b 100644 --- a/tests/docker/dockerfiles/debian-ppc64el-cross.docker +++ b/tests/docker/dockerfiles/debian-ppc64el-cross.docker @@ -1,28 +1,164 @@ +# THIS FILE WAS AUTO-GENERATED # -# Docker ppc64el cross-compiler target +# $ lcitool dockerfile --layers all --cross ppc64le debian-11 qemu # -# This docker target builds on the debian Stretch base image. -# -FROM qemu/debian10 +# https://gitlab.com/libvirt/libvirt-ci -# Add the foreign architecture we want and install dependencies -RUN dpkg --add-architecture ppc64el && \ - apt update && \ - apt install -yy crossbuild-essential-ppc64el +FROM docker.io/library/debian:11-slim -RUN apt update && \ - DEBIAN_FRONTEND=noninteractive eatmydata \ - apt build-dep -yy -a ppc64el --arch-only qemu +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get install -y eatmydata && \ + eatmydata apt-get dist-upgrade -y && \ + eatmydata apt-get install --no-install-recommends -y \ + bash \ + bc \ + bsdextrautils \ + bzip2 \ + ca-certificates \ + ccache \ + dbus \ + debianutils \ + diffutils \ + exuberant-ctags \ + findutils \ + gcovr \ + genisoimage \ + gettext \ + git \ + hostname \ + libpcre2-dev \ + libspice-protocol-dev \ + llvm \ + locales \ + make \ + meson \ + ncat \ + ninja-build \ + openssh-client \ + perl-base \ + pkgconf \ + python3 \ + python3-numpy \ + python3-opencv \ + python3-pillow \ + python3-pip \ + python3-sphinx \ + python3-sphinx-rtd-theme \ + python3-venv \ + python3-yaml \ + rpm2cpio \ + sed \ + sparse \ + tar \ + tesseract-ocr \ + tesseract-ocr-eng \ + texinfo && \ + eatmydata apt-get autoremove -y && \ + eatmydata apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales -# Specify the cross prefix for this image (see tests/docker/common.rc) +ENV LANG "en_US.UTF-8" +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" + +RUN export DEBIAN_FRONTEND=noninteractive && \ + dpkg --add-architecture ppc64el && \ + eatmydata apt-get update && \ + eatmydata apt-get dist-upgrade -y && \ + eatmydata apt-get install --no-install-recommends -y dpkg-dev && \ + eatmydata apt-get install --no-install-recommends -y \ + g++-powerpc64le-linux-gnu \ + gcc-powerpc64le-linux-gnu \ + libaio-dev:ppc64el \ + libasan5:ppc64el \ + libasound2-dev:ppc64el \ + libattr1-dev:ppc64el \ + libbpf-dev:ppc64el \ + libbrlapi-dev:ppc64el \ + libbz2-dev:ppc64el \ + libc6-dev:ppc64el \ + libcacard-dev:ppc64el \ + libcap-ng-dev:ppc64el \ + libcapstone-dev:ppc64el \ + libcurl4-gnutls-dev:ppc64el \ + libdaxctl-dev:ppc64el \ + libdrm-dev:ppc64el \ + libepoxy-dev:ppc64el \ + libfdt-dev:ppc64el \ + libffi-dev:ppc64el \ + libfuse3-dev:ppc64el \ + libgbm-dev:ppc64el \ + libgcrypt20-dev:ppc64el \ + libglib2.0-dev:ppc64el \ + libglusterfs-dev:ppc64el \ + libgnutls28-dev:ppc64el \ + libgtk-3-dev:ppc64el \ + libibumad-dev:ppc64el \ + libibverbs-dev:ppc64el \ + libiscsi-dev:ppc64el \ + libjemalloc-dev:ppc64el \ + libjpeg62-turbo-dev:ppc64el \ + liblttng-ust-dev:ppc64el \ + liblzo2-dev:ppc64el \ + libncursesw5-dev:ppc64el \ + libnfs-dev:ppc64el \ + libnuma-dev:ppc64el \ + libpam0g-dev:ppc64el \ + libpixman-1-dev:ppc64el \ + libpng-dev:ppc64el \ + libpulse-dev:ppc64el \ + librbd-dev:ppc64el \ + librdmacm-dev:ppc64el \ + libsasl2-dev:ppc64el \ + libsdl2-dev:ppc64el \ + libsdl2-image-dev:ppc64el \ + libseccomp-dev:ppc64el \ + libselinux1-dev:ppc64el \ + libslirp-dev:ppc64el \ + libsnappy-dev:ppc64el \ + libspice-server-dev:ppc64el \ + libssh-gcrypt-dev:ppc64el \ + libsystemd-dev:ppc64el \ + libtasn1-6-dev:ppc64el \ + libubsan1:ppc64el \ + libudev-dev:ppc64el \ + liburing-dev:ppc64el \ + libusb-1.0-0-dev:ppc64el \ + libusbredirhost-dev:ppc64el \ + libvdeplug-dev:ppc64el \ + libvirglrenderer-dev:ppc64el \ + libvte-2.91-dev:ppc64el \ + libzstd-dev:ppc64el \ + nettle-dev:ppc64el \ + systemtap-sdt-dev:ppc64el \ + xfslibs-dev:ppc64el \ + zlib1g-dev:ppc64el && \ + eatmydata apt-get autoremove -y && \ + eatmydata apt-get autoclean -y && \ + mkdir -p /usr/local/share/meson/cross && \ + echo "[binaries]\n\ +c = '/usr/bin/powerpc64le-linux-gnu-gcc'\n\ +ar = '/usr/bin/powerpc64le-linux-gnu-gcc-ar'\n\ +strip = '/usr/bin/powerpc64le-linux-gnu-strip'\n\ +pkgconfig = '/usr/bin/powerpc64le-linux-gnu-pkg-config'\n\ +\n\ +[host_machine]\n\ +system = 'linux'\n\ +cpu_family = 'ppc64'\n\ +cpu = 'powerpc64le'\n\ +endian = 'little'" > /usr/local/share/meson/cross/powerpc64le-linux-gnu && \ + dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/powerpc64le-linux-gnu-c++ && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/powerpc64le-linux-gnu-cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/powerpc64le-linux-gnu-g++ && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/powerpc64le-linux-gnu-gcc + +ENV ABI "powerpc64le-linux-gnu" +ENV MESON_OPTS "--cross-file=powerpc64le-linux-gnu" ENV QEMU_CONFIGURE_OPTS --cross-prefix=powerpc64le-linux-gnu- ENV DEF_TARGET_LIST ppc64-softmmu,ppc64-linux-user - -# Install extra libraries to increase code coverage -RUN apt update && \ - DEBIAN_FRONTEND=noninteractive eatmydata \ - apt install -y --no-install-recommends \ - libbz2-dev:ppc64el \ - liblzo2-dev:ppc64el \ - librdmacm-dev:ppc64el \ - libsnappy-dev:ppc64el diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh index 341a07e677..b947bb170d 100755 --- a/tests/lcitool/refresh +++ b/tests/lcitool/refresh @@ -125,6 +125,11 @@ try: trailer=debian_cross_build("mipsel-linux-gnu-", "mipsel-softmmu,mipsel-linux-user")) + generate_dockerfile("debian-ppc64el-cross", "debian-11", + cross="ppc64le", + trailer=debian_cross_build("powerpc64le-linux-gnu-", + "ppc64-softmmu,ppc64-linux-user")) + generate_dockerfile("debian-s390x-cross", "debian-11", cross="s390x", trailer=debian_cross_build("s390x-linux-gnu-", From 9e19fd7d4a569f5802ede460fe03fd2d4d0c295c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alex=20Benn=C3=A9e?= Date: Fri, 27 May 2022 16:35:41 +0100 Subject: [PATCH 11/33] tests/docker: update debian-amd64 with lcitool MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The one minor wrinkle we need to account for is the netmap support still requires building from source. We also include cscope and GNU global as they are used in one of the builds. Signed-off-by: Alex Bennée Cc: Philippe Mathieu-Daudé Cc: Luigi Rizzo Cc: Giuseppe Lettieri Cc: Vincenzo Maffione Reviewed-by: Daniel P. Berrangé Message-Id: <20220527153603.887929-12-alex.bennee@linaro.org> --- .gitlab-ci.d/containers.yml | 3 +- tests/docker/dockerfiles/debian-amd64.docker | 194 ++++++++++++++----- tests/lcitool/refresh | 19 ++ 3 files changed, 164 insertions(+), 52 deletions(-) diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml index e9df90bbdd..be34cbc7ba 100644 --- a/.gitlab-ci.d/containers.yml +++ b/.gitlab-ci.d/containers.yml @@ -14,8 +14,7 @@ amd64-debian11-container: amd64-debian-container: extends: .container_job_template - stage: containers-layer2 - needs: ['amd64-debian10-container'] + stage: containers variables: NAME: debian-amd64 diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-amd64.docker index ed546edcd6..503e282802 100644 --- a/tests/docker/dockerfiles/debian-amd64.docker +++ b/tests/docker/dockerfiles/debian-amd64.docker @@ -1,59 +1,153 @@ +# THIS FILE WAS AUTO-GENERATED # -# Docker x86_64 target +# $ lcitool dockerfile --layers all debian-11 qemu # -# This docker target builds on the Debian Buster base image. Further -# libraries which are not widely available are installed by hand. -# -FROM qemu/debian10 -MAINTAINER Philippe Mathieu-Daudé +# https://gitlab.com/libvirt/libvirt-ci -RUN apt update && \ - DEBIAN_FRONTEND=noninteractive eatmydata \ - apt build-dep -yy qemu +FROM docker.io/library/debian:11-slim -RUN apt update && \ - DEBIAN_FRONTEND=noninteractive eatmydata \ - apt install -y --no-install-recommends \ - cscope \ - genisoimage \ - exuberant-ctags \ - global \ - libbz2-dev \ - liblzo2-dev \ - libgcrypt20-dev \ - libfdt-dev \ - librdmacm-dev \ - libsasl2-dev \ - libsnappy-dev \ - libvte-dev \ - netcat-openbsd \ - openssh-client \ - python3-numpy \ - python3-opencv \ - python3-venv +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get install -y eatmydata && \ + eatmydata apt-get dist-upgrade -y && \ + eatmydata apt-get install --no-install-recommends -y \ + bash \ + bc \ + bsdextrautils \ + bzip2 \ + ca-certificates \ + ccache \ + clang \ + dbus \ + debianutils \ + diffutils \ + exuberant-ctags \ + findutils \ + g++ \ + gcc \ + gcovr \ + genisoimage \ + gettext \ + git \ + hostname \ + libaio-dev \ + libasan5 \ + libasound2-dev \ + libattr1-dev \ + libbpf-dev \ + libbrlapi-dev \ + libbz2-dev \ + libc6-dev \ + libcacard-dev \ + libcap-ng-dev \ + libcapstone-dev \ + libcurl4-gnutls-dev \ + libdaxctl-dev \ + libdrm-dev \ + libepoxy-dev \ + libfdt-dev \ + libffi-dev \ + libfuse3-dev \ + libgbm-dev \ + libgcrypt20-dev \ + libglib2.0-dev \ + libglusterfs-dev \ + libgnutls28-dev \ + libgtk-3-dev \ + libibumad-dev \ + libibverbs-dev \ + libiscsi-dev \ + libjemalloc-dev \ + libjpeg62-turbo-dev \ + liblttng-ust-dev \ + liblzo2-dev \ + libncursesw5-dev \ + libnfs-dev \ + libnuma-dev \ + libpam0g-dev \ + libpcre2-dev \ + libpixman-1-dev \ + libpmem-dev \ + libpng-dev \ + libpulse-dev \ + librbd-dev \ + librdmacm-dev \ + libsasl2-dev \ + libsdl2-dev \ + libsdl2-image-dev \ + libseccomp-dev \ + libselinux1-dev \ + libslirp-dev \ + libsnappy-dev \ + libspice-protocol-dev \ + libspice-server-dev \ + libssh-gcrypt-dev \ + libsystemd-dev \ + libtasn1-6-dev \ + libubsan1 \ + libudev-dev \ + liburing-dev \ + libusb-1.0-0-dev \ + libusbredirhost-dev \ + libvdeplug-dev \ + libvirglrenderer-dev \ + libvte-2.91-dev \ + libxen-dev \ + libzstd-dev \ + llvm \ + locales \ + make \ + meson \ + multipath-tools \ + ncat \ + nettle-dev \ + ninja-build \ + openssh-client \ + perl-base \ + pkgconf \ + python3 \ + python3-numpy \ + python3-opencv \ + python3-pillow \ + python3-pip \ + python3-sphinx \ + python3-sphinx-rtd-theme \ + python3-venv \ + python3-yaml \ + rpm2cpio \ + sed \ + sparse \ + systemtap-sdt-dev \ + tar \ + tesseract-ocr \ + tesseract-ocr-eng \ + texinfo \ + xfslibs-dev \ + zlib1g-dev && \ + eatmydata apt-get autoremove -y && \ + eatmydata apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales && \ + dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/c++ && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/clang && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/g++ && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc -# virgl -RUN apt update && \ - DEBIAN_FRONTEND=noninteractive eatmydata \ - apt install -y --no-install-recommends \ - libegl1-mesa-dev \ - libepoxy-dev \ - libgbm-dev -RUN git clone https://gitlab.freedesktop.org/virgl/virglrenderer.git /usr/src/virglrenderer && \ - cd /usr/src/virglrenderer && git checkout virglrenderer-0.8.0 -RUN cd /usr/src/virglrenderer && ./autogen.sh && ./configure --disable-tests && make install - -# netmap -RUN apt update && \ - DEBIAN_FRONTEND=noninteractive eatmydata \ - apt install -y --no-install-recommends \ - linux-headers-amd64 +ENV LANG "en_US.UTF-8" +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" +# netmap/cscope/global +RUN DEBIAN_FRONTEND=noninteractive eatmydata \ + apt install -y --no-install-recommends \ + cscope\ + global\ + linux-headers-amd64 RUN git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap RUN cd /usr/src/netmap && git checkout v11.3 RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-amd64) && make install ENV QEMU_CONFIGURE_OPTS --enable-netmap - -RUN ldconfig - -# gcrypt -ENV QEMU_CONFIGURE_OPTS $QEMU_CONFIGURE_OPTS --enable-gcrypt diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh index b947bb170d..5e260f8cd6 100755 --- a/tests/lcitool/refresh +++ b/tests/lcitool/refresh @@ -76,6 +76,23 @@ ubuntu2004_tsanhack = [ ] +# Netmap still needs to be manually built as it is yet to be packaged +# into a distro. We also add cscope and gtags which are used in the CI +# test +debian11_extras = [ + "# netmap/cscope/global\n", + "RUN DEBIAN_FRONTEND=noninteractive eatmydata \\\n", + " apt install -y --no-install-recommends \\\n", + " cscope\\\n", + " global\\\n", + " linux-headers-amd64\n", + "RUN git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap\n", + "RUN cd /usr/src/netmap && git checkout v11.3\n", + "RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-amd64) && make install\n", + "ENV QEMU_CONFIGURE_OPTS --enable-netmap\n" +] + + def debian_cross_build(prefix, targets): conf = "ENV QEMU_CONFIGURE_OPTS --cross-prefix=%s\n" % (prefix) targets = "ENV DEF_TARGET_LIST %s\n" % (targets) @@ -92,6 +109,8 @@ try: # generate_dockerfile("alpine", "alpine-edge") generate_dockerfile("centos8", "centos-stream-8") + generate_dockerfile("debian-amd64", "debian-11", + trailer="".join(debian11_extras)) generate_dockerfile("fedora", "fedora-35") generate_dockerfile("opensuse-leap", "opensuse-leap-152") generate_dockerfile("ubuntu2004", "ubuntu-2004", From ea857a3b7976010a7fe7b59a89941964028f502a Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Fri, 27 May 2022 16:35:42 +0100 Subject: [PATCH 12/33] configure: do not define or use the CPP variable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Just hardcode $(CC) -E, it should be enough. Signed-off-by: Paolo Bonzini Reviewed-by: Richard Henderson Message-Id: <20220517092616.1272238-2-pbonzini@redhat.com> Signed-off-by: Alex Bennée Message-Id: <20220527153603.887929-13-alex.bennee@linaro.org> --- configure | 3 --- pc-bios/optionrom/Makefile | 2 +- 2 files changed, 1 insertion(+), 4 deletions(-) diff --git a/configure b/configure index f2baf2f526..c88ef94fec 100755 --- a/configure +++ b/configure @@ -376,7 +376,6 @@ fi ar="${AR-${cross_prefix}ar}" as="${AS-${cross_prefix}as}" ccas="${CCAS-$cc}" -cpp="${CPP-$cc -E}" objcopy="${OBJCOPY-${cross_prefix}objcopy}" ld="${LD-${cross_prefix}ld}" ranlib="${RANLIB-${cross_prefix}ranlib}" @@ -2014,7 +2013,6 @@ echo "CC=$cc" >> $config_host_mak echo "AR=$ar" >> $config_host_mak echo "AS=$as" >> $config_host_mak echo "CCAS=$ccas" >> $config_host_mak -echo "CPP=$cpp" >> $config_host_mak echo "OBJCOPY=$objcopy" >> $config_host_mak echo "LD=$ld" >> $config_host_mak echo "QEMU_CFLAGS=$QEMU_CFLAGS" >> $config_host_mak @@ -2257,7 +2255,6 @@ preserve_env() { preserve_env AR preserve_env AS preserve_env CC -preserve_env CPP preserve_env CFLAGS preserve_env CXX preserve_env CXXFLAGS diff --git a/pc-bios/optionrom/Makefile b/pc-bios/optionrom/Makefile index 2494ad9c25..17ccc76241 100644 --- a/pc-bios/optionrom/Makefile +++ b/pc-bios/optionrom/Makefile @@ -50,7 +50,7 @@ override LDFLAGS = -m $(LD_I386_EMULATION) -T $(SRC_DIR)/flat.lds pvh.img: pvh.o pvh_main.o %.o: %.S - $(call quiet-command,$(CPP) $(CPPFLAGS) -c -o - $< | $(AS) $(ASFLAGS) -o $@,"AS","$@") + $(call quiet-command,$(CC) $(CPPFLAGS) -E -o - $< | $(AS) $(ASFLAGS) -o $@,"AS","$@") %.o: %.c $(call quiet-command,$(CC) $(CPPFLAGS) $(CFLAGS) -c $< -o $@,"CC","$@") From 231ca96316f17d0af8c5e72cc2c496c68ee23ab6 Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Fri, 27 May 2022 16:35:43 +0100 Subject: [PATCH 13/33] build: clean up ninja invocation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fix an incorrect "@@:" and move "-d keepdepfile" to the NINJAFLAGS variable. Signed-off-by: Paolo Bonzini Reviewed-by: Richard Henderson Message-Id: <20220517092616.1272238-3-pbonzini@redhat.com> Signed-off-by: Alex Bennée Message-Id: <20220527153603.887929-14-alex.bennee@linaro.org> --- Makefile | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index b842dbccdb..fad312040f 100644 --- a/Makefile +++ b/Makefile @@ -143,7 +143,7 @@ MAKE.q = $(findstring q,$(firstword $(filter-out --%,$(MAKEFLAGS)))) MAKE.nq = $(if $(word 2, $(MAKE.n) $(MAKE.q)),nq) NINJAFLAGS = $(if $V,-v) $(if $(MAKE.n), -n) $(if $(MAKE.k), -k0) \ $(filter-out -j, $(lastword -j1 $(filter -l% -j%, $(MAKEFLAGS)))) \ - + -d keepdepfile ninja-cmd-goals = $(or $(MAKECMDGOALS), all) ninja-cmd-goals += $(foreach t, $(.check.build-suites), $(.check-$t.deps)) ninja-cmd-goals += $(foreach t, $(.bench.build-suites), $(.bench-$t.deps)) @@ -160,8 +160,8 @@ $(ninja-targets): run-ninja # --output-sync line. run-ninja: config-host.mak ifneq ($(filter $(ninja-targets), $(ninja-cmd-goals)),) - +$(quiet-@)$(if $(MAKE.nq),@:, $(NINJA) -d keepdepfile \ - $(NINJAFLAGS) $(sort $(filter $(ninja-targets), $(ninja-cmd-goals))) | cat) + +$(if $(MAKE.nq),@:,$(quiet-@)$(NINJA) $(NINJAFLAGS) \ + $(sort $(filter $(ninja-targets), $(ninja-cmd-goals))) | cat) endif endif From 9b32ba53f36428abdecc79a0a047c2636af130d1 Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Fri, 27 May 2022 16:35:44 +0100 Subject: [PATCH 14/33] build: add a more generic way to specify make->ninja dependencies MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Let any make target specify ninja goals that needs to be built for it (though selecting the goals is _not_ recursive on depending targets) instead of having a custom mechanism only for "make check" and "make bench". Signed-off-by: Paolo Bonzini Message-Id: <20220517092616.1272238-4-pbonzini@redhat.com> Signed-off-by: Alex Bennée Message-Id: <20220527153603.887929-15-alex.bennee@linaro.org> --- Makefile | 3 +-- scripts/mtest2make.py | 8 ++++---- 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/Makefile b/Makefile index fad312040f..3c0d89057e 100644 --- a/Makefile +++ b/Makefile @@ -145,8 +145,7 @@ NINJAFLAGS = $(if $V,-v) $(if $(MAKE.n), -n) $(if $(MAKE.k), -k0) \ $(filter-out -j, $(lastword -j1 $(filter -l% -j%, $(MAKEFLAGS)))) \ -d keepdepfile ninja-cmd-goals = $(or $(MAKECMDGOALS), all) -ninja-cmd-goals += $(foreach t, $(.check.build-suites), $(.check-$t.deps)) -ninja-cmd-goals += $(foreach t, $(.bench.build-suites), $(.bench-$t.deps)) +ninja-cmd-goals += $(foreach g, $(MAKECMDGOALS), $(.ninja-goals.$g)))) makefile-targets := build.ninja ctags TAGS cscope dist clean uninstall # "ninja -t targets" also lists all prerequisites. If build system diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py index 304634b71e..0fe81efbbc 100644 --- a/scripts/mtest2make.py +++ b/scripts/mtest2make.py @@ -81,12 +81,12 @@ def emit_prolog(suites, prefix): def emit_suite_deps(name, suite, prefix): deps = ' '.join(suite.deps) - targets = f'{prefix}-{name} {prefix}-report-{name}.junit.xml {prefix} {prefix}-report.junit.xml' + targets = [f'{prefix}-{name}', f'{prefix}-report-{name}.junit.xml', f'{prefix}', f'{prefix}-report.junit.xml', + f'{prefix}-build'] print() print(f'.{prefix}-{name}.deps = {deps}') - print(f'ifneq ($(filter {prefix}-build {targets}, $(MAKECMDGOALS)),)') - print(f'.{prefix}.build-suites += {name}') - print(f'endif') + for t in targets: + print(f'.ninja-goals.{t} += $(.{prefix}-{name}.deps)') def emit_suite(name, suite, prefix): emit_suite_deps(name, suite, prefix) From 95808875be96e17b885a9e4f050897c67e8b9aaf Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Fri, 27 May 2022 16:35:45 +0100 Subject: [PATCH 15/33] build: do a full build before running TCG tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TCG tests need both QEMU and firmware to be built, so do "ninja all" before trying to run them. Signed-off-by: Paolo Bonzini Reviewed-by: Richard Henderson Message-Id: <20220517092616.1272238-5-pbonzini@redhat.com> Signed-off-by: Alex Bennée Message-Id: <20220527153603.887929-16-alex.bennee@linaro.org> --- tests/Makefile.include | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/Makefile.include b/tests/Makefile.include index ec84b2ebc0..72ce0561f4 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -57,7 +57,7 @@ $(TCG_TESTS_TARGETS:%=build-tcg-tests-%): build-tcg-tests-%: $(BUILD_DIR)/tests/ "BUILD","$* guest-tests") .PHONY: $(TCG_TESTS_TARGETS:%=run-tcg-tests-%) -$(TCG_TESTS_TARGETS:%=run-tcg-tests-%): run-tcg-tests-%: build-tcg-tests-% $(if $(CONFIG_PLUGIN),test-plugins) +$(TCG_TESTS_TARGETS:%=run-tcg-tests-%): run-tcg-tests-%: build-tcg-tests-% $(call quiet-command, \ $(MAKE) -C tests/tcg/$* -f ../Makefile.target $(SUBDIR_MAKEFLAGS) \ TARGET="$*" SRC_PATH="$(SRC_PATH)" SPEED=$(SPEED) run, \ @@ -74,6 +74,7 @@ $(TCG_TESTS_TARGETS:%=clean-tcg-tests-%): clean-tcg-tests-%: build-tcg: $(BUILD_TCG_TARGET_RULES) .PHONY: check-tcg +.ninja-goals.check-tcg = all $(if $(CONFIG_PLUGIN),test-plugins) check-tcg: $(RUN_TCG_TARGET_RULES) .PHONY: clean-tcg From 45f1eecdd63f9e4fa93fef01dd826e7706ac6d7b Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Fri, 27 May 2022 16:35:46 +0100 Subject: [PATCH 16/33] configure, meson: move symlinking of ROMs to meson MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This is useful because pc-bios/meson.build already has a list of all ROM files, and thus does not need to use wildcards. The problems with wildcards are mentioned above the definition of the LINKS variable, but then the recommendation is disattended. Reviewed-by: Richard Henderson Signed-off-by: Paolo Bonzini Message-Id: <20220517092616.1272238-6-pbonzini@redhat.com> Signed-off-by: Alex Bennée Message-Id: <20220527153603.887929-17-alex.bennee@linaro.org> --- configure | 15 --------------- pc-bios/meson.build | 17 ++++++++++++----- 2 files changed, 12 insertions(+), 20 deletions(-) diff --git a/configure b/configure index c88ef94fec..7b6adc29fe 100755 --- a/configure +++ b/configure @@ -2100,21 +2100,6 @@ LINKS="$LINKS tests/avocado tests/data" LINKS="$LINKS tests/qemu-iotests/check" LINKS="$LINKS python" LINKS="$LINKS contrib/plugins/Makefile " -for bios_file in \ - $source_path/pc-bios/*.bin \ - $source_path/pc-bios/*.elf \ - $source_path/pc-bios/*.lid \ - $source_path/pc-bios/*.rom \ - $source_path/pc-bios/*.dtb \ - $source_path/pc-bios/*.img \ - $source_path/pc-bios/openbios-* \ - $source_path/pc-bios/u-boot.* \ - $source_path/pc-bios/palcode-* \ - $source_path/pc-bios/qemu_vga.ndrv - -do - LINKS="$LINKS pc-bios/$(basename $bios_file)" -done for f in $LINKS ; do if [ -e "$source_path/$f" ]; then mkdir -p `dirname ./$f` diff --git a/pc-bios/meson.build b/pc-bios/meson.build index c86dedf7df..41ba1c0ec7 100644 --- a/pc-bios/meson.build +++ b/pc-bios/meson.build @@ -23,7 +23,7 @@ if unpack_edk2_blobs endforeach endif -blobs = files( +blobs = [ 'bios.bin', 'bios-256k.bin', 'bios-microvm.bin', @@ -83,11 +83,18 @@ blobs = files( 'npcm7xx_bootrom.bin', 'vof.bin', 'vof-nvram.bin', -) +] -if get_option('install_blobs') - install_data(blobs, install_dir: qemu_datadir) -endif +ln_s = [find_program('ln', required: true), '-sf'] +foreach f : blobs + roms += custom_target(f, + build_by_default: have_system, + output: f, + input: files('meson.build'), # dummy input + install: get_option('install_blobs'), + install_dir: qemu_datadir, + command: [ ln_s, meson.project_source_root() / 'pc-bios' / f, '@OUTPUT@' ]) +endforeach subdir('descriptors') subdir('keymaps') From 867998cf5680329a150cb6d51a15d6990be49c77 Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Fri, 27 May 2022 16:35:47 +0100 Subject: [PATCH 17/33] tests/tcg: correct target CPU for sparc32 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit We do not want v8plus for pure sparc32, as the difference with the V8 ABI are only meaningful on 64-bit CPUs suh as ultrasparc; supersparc is the best CPU to use for 32-bit. Signed-off-by: Paolo Bonzini Reviewed-by: Richard Henderson Message-Id: <20220517092616.1272238-7-pbonzini@redhat.com> Signed-off-by: Alex Bennée Message-Id: <20220527153603.887929-18-alex.bennee@linaro.org> --- tests/tcg/configure.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/tcg/configure.sh b/tests/tcg/configure.sh index 691d90abac..59f2403d1a 100755 --- a/tests/tcg/configure.sh +++ b/tests/tcg/configure.sh @@ -70,7 +70,7 @@ fi : ${cross_cc_riscv64="riscv64-linux-gnu-gcc"} : ${cross_cc_s390x="s390x-linux-gnu-gcc"} : ${cross_cc_sh4="sh4-linux-gnu-gcc"} -: ${cross_cc_cflags_sparc="-m32 -mv8plus -mcpu=ultrasparc"} +: ${cross_cc_cflags_sparc="-m32 -mcpu=supersparc"} : ${cross_cc_sparc64="sparc64-linux-gnu-gcc"} : ${cross_cc_cflags_sparc64="-m64 -mcpu=ultrasparc"} : ${cross_cc_x86_64="x86_64-linux-gnu-gcc"} From cd362defbbd09cbbc08b3bb465141542887b8cef Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Fri, 27 May 2022 16:35:48 +0100 Subject: [PATCH 18/33] tests/tcg: merge configure.sh back into main configure script MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit tests/tcg/configure.sh has a complicated story. In the beginning its code ran as part of the creation of config-target.mak files, and that is where it placed the information on the target compiler. However, probing for the buildability of TCG tests required multiple inclusions of config-target.mak in the _main_ Makefile (not in Makefile.target, which took care of building the QEMU executables in the pre-Meson era), which polluted the namespace. Thus, it was moved to a separate directory. It created small config-*.mak files in $(BUILD_DIR)/tests/tcg. Those were also included multiple times, but at least they were small and manageable; this was also an important step in disentangling the TCG tests from Makefile.target. Since then, Meson has allowed the configure script to go on a diet. A few compilation tests survive (mostly for sanitizers) but these days it mostly takes care of command line parsing, looking for tools, and setting up the environment for Meson to do its stuff. It's time to extend configure with the capability to build for more than just one target: not just tests, but also firmware. As a first step, integrate all the logic to find cross compilers in the configure script, and move tests/tcg/configure.sh back there (though as a separate loop, not integrated in the one that generates target configurations for Meson). tests/tcg is actually very close to being buildable as a standalone project, so I actually expect the compiler tests to move back to tests/tcg, as a "configure" script of sorts which would run at Make time after the docker images are built. The GCC tree has a similar idea of doing only bare-bones tree-wide configuration and leaving the rest for Make time. Signed-off-by: Paolo Bonzini Acked-by: Richard Henderson Message-Id: <20220517092616.1272238-8-pbonzini@redhat.com> Signed-off-by: Alex Bennée Message-Id: <20220527153603.887929-19-alex.bennee@linaro.org> --- configure | 398 +++++++++++++++++++++++++++++++++++++++-- tests/Makefile.include | 1 - tests/tcg/configure.sh | 376 -------------------------------------- 3 files changed, 388 insertions(+), 387 deletions(-) delete mode 100755 tests/tcg/configure.sh diff --git a/configure b/configure index 7b6adc29fe..f91ac632e7 100755 --- a/configure +++ b/configure @@ -109,6 +109,20 @@ error_exit() { } do_compiler() { + # Run the compiler, capturing its output to the log. First argument + # is compiler binary to execute. + local compiler="$1" + shift + if test -n "$BASH_VERSION"; then eval ' + echo >>config.log " +funcs: ${FUNCNAME[*]} +lines: ${BASH_LINENO[*]}" + '; fi + echo $compiler "$@" >> config.log + $compiler "$@" >> config.log 2>&1 || return $? +} + +do_compiler_werror() { # Run the compiler, capturing its output to the log. First argument # is compiler binary to execute. compiler="$1" @@ -142,15 +156,15 @@ lines: ${BASH_LINENO[*]}" } do_cc() { - do_compiler "$cc" $CPU_CFLAGS "$@" + do_compiler_werror "$cc" $CPU_CFLAGS "$@" } do_cxx() { - do_compiler "$cxx" $CPU_CFLAGS "$@" + do_compiler_werror "$cxx" $CPU_CFLAGS "$@" } do_objc() { - do_compiler "$objcc" $CPU_CFLAGS "$@" + do_compiler_werror "$objcc" $CPU_CFLAGS "$@" } # Append $2 to the variable named $1, with space separation @@ -345,11 +359,9 @@ for opt do ;; --cross-cc-cflags-*) cc_arch=${opt#--cross-cc-cflags-}; cc_arch=${cc_arch%%=*} eval "cross_cc_cflags_${cc_arch}=\$optarg" - cross_cc_vars="$cross_cc_vars cross_cc_cflags_${cc_arch}" ;; --cross-cc-*) cc_arch=${opt#--cross-cc-}; cc_arch=${cc_arch%%=*} eval "cross_cc_${cc_arch}=\$optarg" - cross_cc_vars="$cross_cc_vars cross_cc_${cc_arch}" ;; esac done @@ -944,7 +956,6 @@ esac if eval test -z "\${cross_cc_$cpu}"; then eval "cross_cc_${cpu}=\$cc" - cross_cc_vars="$cross_cc_vars cross_cc_${cpu}" fi default_target_list="" @@ -1800,6 +1811,248 @@ case "$slirp" in ;; esac +########################################## +# functions to probe cross compilers + +container="no" +if test $use_containers = "yes"; then + if has "docker" || has "podman"; then + container=$($python $source_path/tests/docker/docker.py probe) + fi +fi + +# cross compilers defaults, can be overridden with --cross-cc-ARCH +: ${cross_cc_aarch64="aarch64-linux-gnu-gcc"} +: ${cross_cc_aarch64_be="$cross_cc_aarch64"} +: ${cross_cc_cflags_aarch64_be="-mbig-endian"} +: ${cross_cc_alpha="alpha-linux-gnu-gcc"} +: ${cross_cc_arm="arm-linux-gnueabihf-gcc"} +: ${cross_cc_cflags_armeb="-mbig-endian"} +: ${cross_cc_hexagon="hexagon-unknown-linux-musl-clang"} +: ${cross_cc_cflags_hexagon="-mv67 -O2 -static"} +: ${cross_cc_hppa="hppa-linux-gnu-gcc"} +: ${cross_cc_i386="i686-linux-gnu-gcc"} +: ${cross_cc_cflags_i386="-m32"} +: ${cross_cc_m68k="m68k-linux-gnu-gcc"} +: ${cross_cc_microblaze="microblaze-linux-musl-gcc"} +: ${cross_cc_mips64el="mips64el-linux-gnuabi64-gcc"} +: ${cross_cc_mips64="mips64-linux-gnuabi64-gcc"} +: ${cross_cc_mipsel="mipsel-linux-gnu-gcc"} +: ${cross_cc_mips="mips-linux-gnu-gcc"} +: ${cross_cc_nios2="nios2-linux-gnu-gcc"} +: ${cross_cc_ppc="powerpc-linux-gnu-gcc"} +: ${cross_cc_cflags_ppc="-m32"} +: ${cross_cc_ppc64="powerpc64-linux-gnu-gcc"} +: ${cross_cc_cflags_ppc64="-m64 -mbig-endian"} +: ${cross_cc_ppc64le="$cross_cc_ppc64"} +: ${cross_cc_cflags_ppc64le="-m64 -mlittle-endian"} +: ${cross_cc_riscv64="riscv64-linux-gnu-gcc"} +: ${cross_cc_s390x="s390x-linux-gnu-gcc"} +: ${cross_cc_sh4="sh4-linux-gnu-gcc"} +: ${cross_cc_cflags_sparc="-m32 -mcpu=supersparc"} +: ${cross_cc_sparc64="sparc64-linux-gnu-gcc"} +: ${cross_cc_cflags_sparc64="-m64 -mcpu=ultrasparc"} +: ${cross_cc_x86_64="x86_64-linux-gnu-gcc"} +: ${cross_cc_cflags_x86_64="-m64"} + +# tricore is special as it doesn't have a compiler +: ${cross_as_tricore="tricore-as"} +: ${cross_ld_tricore="tricore-ld"} + +probe_target_compiler() { + # reset all output variables + container_image= + container_hosts= + container_cross_cc= + container_cross_as= + container_cross_ld= + target_cc= + target_as= + target_ld= + + case $1 in + aarch64) container_hosts="x86_64 aarch64" ;; + alpha) container_hosts=x86_64 ;; + arm) container_hosts="x86_64 aarch64" ;; + cris) container_hosts=x86_64 ;; + hexagon) container_hosts=x86_64 ;; + hppa) container_hosts=x86_64 ;; + i386) container_hosts=x86_64 ;; + m68k) container_hosts=x86_64 ;; + microblaze) container_hosts=x86_64 ;; + mips64el) container_hosts=x86_64 ;; + mips64) container_hosts=x86_64 ;; + mipsel) container_hosts=x86_64 ;; + mips) container_hosts=x86_64 ;; + nios2) container_hosts=x86_64 ;; + ppc) container_hosts=x86_64 ;; + ppc64|ppc64le) container_hosts=x86_64 ;; + riscv64) container_hosts=x86_64 ;; + s390x) container_hosts=x86_64 ;; + sh4) container_hosts=x86_64 ;; + sparc64) container_hosts=x86_64 ;; + tricore) container_hosts=x86_64 ;; + x86_64) container_hosts="aarch64 ppc64el x86_64" ;; + xtensa*) container_hosts=x86_64 ;; + esac + + for host in $container_hosts; do + test "$container" != no || continue + test "$host" = "$cpu" || continue + case $1 in + aarch64) + # We don't have any bigendian build tools so we only use this for AArch64 + container_image=debian-arm64-cross + container_cross_cc=aarch64-linux-gnu-gcc-10 + ;; + alpha) + container_image=debian-alpha-cross + container_cross_cc=alpha-linux-gnu-gcc + ;; + arm) + # We don't have any bigendian build tools so we only use this for ARM + container_image=debian-armhf-cross + container_cross_cc=arm-linux-gnueabihf-gcc + ;; + cris) + container_image=fedora-cris-cross + container_cross_cc=cris-linux-gnu-gcc + ;; + hexagon) + container_image=debian-hexagon-cross + container_cross_cc=hexagon-unknown-linux-musl-clang + ;; + hppa) + container_image=debian-hppa-cross + container_cross_cc=hppa-linux-gnu-gcc + ;; + i386) + container_image=fedora-i386-cross + container_cross_cc=gcc + ;; + m68k) + container_image=debian-m68k-cross + container_cross_cc=m68k-linux-gnu-gcc + ;; + microblaze) + container_image=debian-microblaze-cross + container_cross_cc=microblaze-linux-musl-gcc + ;; + mips64el) + container_image=debian-mips64el-cross + container_cross_cc=mips64el-linux-gnuabi64-gcc + ;; + mips64) + container_image=debian-mips64-cross + container_cross_cc=mips64-linux-gnuabi64-gcc + ;; + mipsel) + container_image=debian-mipsel-cross + container_cross_cc=mipsel-linux-gnu-gcc + ;; + mips) + container_image=debian-mips-cross + container_cross_cc=mips-linux-gnu-gcc + ;; + nios2) + container_image=debian-nios2-cross + container_cross_cc=nios2-linux-gnu-gcc + ;; + ppc) + container_image=debian-powerpc-test-cross + container_cross_cc=powerpc-linux-gnu-gcc-10 + ;; + ppc64|ppc64le) + container_image=debian-powerpc-test-cross + container_cross_cc=powerpc${1#ppc}-linux-gnu-gcc-10 + ;; + riscv64) + container_image=debian-riscv64-test-cross + container_cross_cc=riscv64-linux-gnu-gcc + ;; + s390x) + container_image=debian-s390x-cross + container_cross_cc=s390x-linux-gnu-gcc + ;; + sh4) + container_image=debian-sh4-cross + container_cross_cc=sh4-linux-gnu-gcc + ;; + sparc64) + container_image=debian-sparc64-cross + container_cross_cc=sparc64-linux-gnu-gcc + ;; + tricore) + container_image=debian-tricore-cross + container_cross_as=tricore-as + container_cross_ld=tricore-ld + ;; + x86_64) + container_image=debian-amd64-cross + container_cross_cc=x86_64-linux-gnu-gcc + ;; + xtensa*) + # FIXME: xtensa-linux-user? + container_hosts=x86_64 + container_image=debian-xtensa-cross + + # default to the dc232b cpu + container_cross_cc=/opt/2020.07/xtensa-dc232b-elf/bin/xtensa-dc232b-elf-gcc + ;; + esac + done + + eval "target_cflags=\${cross_cc_cflags_$1}" + if eval test -n "\"\${cross_cc_$1}\""; then + if eval has "\"\${cross_cc_$1}\""; then + eval "target_cc=\"\${cross_cc_$1}\"" + case $1 in + i386|x86_64) + if $target_cc --version | grep -qi "clang"; then + unset target_cc + fi + ;; + esac + fi + fi + if eval test -n "\"\${cross_as_$1}\""; then + if eval has "\"\${cross_as_$1}\""; then + eval "target_as=\"\${cross_as_$1}\"" + fi + fi + if eval test -n "\"\${cross_ld_$1}\""; then + if eval has "\"\${cross_ld_$1}\""; then + eval "target_ld=\"\${cross_ld_$1}\"" + fi + fi +} + +write_target_makefile() { + if test -n "$target_cc"; then + echo "CC=$target_cc" + fi + if test -n "$target_as"; then + echo "AS=$target_as" + fi + if test -n "$target_ld"; then + echo "LD=$target_ld" + fi +} + +write_container_target_makefile() { + if test -n "$container_cross_cc"; then + echo "CC=\$(DOCKER_SCRIPT) cc --cc $container_cross_cc -i qemu/$container_image -s $source_path --" + fi + if test -n "$container_cross_as"; then + echo "AS=\$(DOCKER_SCRIPT) cc --cc $container_cross_as -i qemu/$container_image -s $source_path --" + fi + if test -n "$container_cross_ld"; then + echo "LD=\$(DOCKER_SCRIPT) cc --cc $container_cross_ld -i qemu/$container_image -s $source_path --" + fi +} + + + ########################################## # End of CC checks # After here, no more $cc or $ld runs @@ -2107,11 +2360,136 @@ for f in $LINKS ; do fi done -(for i in $cross_cc_vars; do - export $i +# tests/tcg configuration +(makefile=tests/tcg/Makefile.prereqs +echo "# Automatically generated by configure - do not modify" > $makefile + +config_host_mak=tests/tcg/config-host.mak +echo "# Automatically generated by configure - do not modify" > $config_host_mak +echo "SRC_PATH=$source_path" >> $config_host_mak +echo "HOST_CC=$host_cc" >> $config_host_mak + +tcg_tests_targets= +for target in $target_list; do + arch=${target%%-*} + + probe_target_compiler ${arch} + config_target_mak=tests/tcg/config-$target.mak + + echo "# Automatically generated by configure - do not modify" > $config_target_mak + echo "TARGET_NAME=$arch" >> $config_target_mak + case $target in + *-softmmu) + test -f $source_path/tests/tcg/$arch/Makefile.softmmu-target || continue + qemu="qemu-system-$arch" + ;; + *-linux-user|*-bsd-user) + qemu="qemu-$arch" + ;; + esac + + got_cross_cc=no + unset build_static + + if test -n "$target_cc"; then + write_c_skeleton + if ! do_compiler "$target_cc" $target_cflags \ + -o $TMPE $TMPC -static ; then + # For host systems we might get away with building without -static + if do_compiler "$target_cc" $target_cflags \ + -o $TMPE $TMPC ; then + got_cross_cc=yes + fi + else + got_cross_cc=yes + build_static=y + fi + elif test -n "$target_as" && test -n "$target_ld"; then + # Special handling for assembler only tests + case $target in + tricore-softmmu) got_cross_cc=yes ;; + esac + fi + + if test $got_cross_cc = yes; then + # Test for compiler features for optional tests. We only do this + # for cross compilers because ensuring the docker containers based + # compilers is a requirememt for adding a new test that needs a + # compiler feature. + + echo "BUILD_STATIC=$build_static" >> $config_target_mak + write_target_makefile >> $config_target_mak + case $target in + aarch64-*) + if do_compiler "$target_cc" $target_cflags \ + -march=armv8.1-a+sve -o $TMPE $TMPC; then + echo "CROSS_CC_HAS_SVE=y" >> $config_target_mak + fi + if do_compiler "$target_cc" $target_cflags \ + -march=armv8.1-a+sve2 -o $TMPE $TMPC; then + echo "CROSS_CC_HAS_SVE2=y" >> $config_target_mak + fi + if do_compiler "$target_cc" $target_cflags \ + -march=armv8.3-a -o $TMPE $TMPC; then + echo "CROSS_CC_HAS_ARMV8_3=y" >> $config_target_mak + fi + if do_compiler "$target_cc" $target_cflags \ + -mbranch-protection=standard -o $TMPE $TMPC; then + echo "CROSS_CC_HAS_ARMV8_BTI=y" >> $config_target_mak + fi + if do_compiler "$target_cc" $target_cflags \ + -march=armv8.5-a+memtag -o $TMPE $TMPC; then + echo "CROSS_CC_HAS_ARMV8_MTE=y" >> $config_target_mak + fi + ;; + ppc*) + if do_compiler "$target_cc" $target_cflags \ + -mpower8-vector -o $TMPE $TMPC; then + echo "CROSS_CC_HAS_POWER8_VECTOR=y" >> $config_target_mak + fi + if do_compiler "$target_cc" $target_cflags \ + -mpower10 -o $TMPE $TMPC; then + echo "CROSS_CC_HAS_POWER10=y" >> $config_target_mak + fi + ;; + i386-linux-user) + if do_compiler "$target_cc" $target_cflags \ + -Werror -fno-pie -o $TMPE $TMPC; then + echo "CROSS_CC_HAS_I386_NOPIE=y" >> $config_target_mak + fi + ;; + esac + elif test -n "$container_image"; then + echo "build-tcg-tests-$target: docker-image-$container_image" >> $makefile + echo "BUILD_STATIC=y" >> $config_target_mak + write_container_target_makefile >> $config_target_mak + case $target in + aarch64-*) + echo "CROSS_CC_HAS_SVE=y" >> $config_target_mak + echo "CROSS_CC_HAS_SVE2=y" >> $config_target_mak + echo "CROSS_CC_HAS_ARMV8_3=y" >> $config_target_mak + echo "CROSS_CC_HAS_ARMV8_BTI=y" >> $config_target_mak + echo "CROSS_CC_HAS_ARMV8_MTE=y" >> $config_target_mak + ;; + ppc*) + echo "CROSS_CC_HAS_POWER8_VECTOR=y" >> $config_target_mak + echo "CROSS_CC_HAS_POWER10=y" >> $config_target_mak + ;; + i386-linux-user) + echo "CROSS_CC_HAS_I386_NOPIE=y" >> $config_target_mak + ;; + esac + got_cross_cc=yes + fi + if test $got_cross_cc = yes; then + mkdir -p tests/tcg/$target + echo "QEMU=$PWD/$qemu" >> $config_target_mak + echo "EXTRA_CFLAGS=$target_cflags" >> $config_target_mak + echo "run-tcg-tests-$target: $qemu\$(EXESUF)" >> $makefile + tcg_tests_targets="$tcg_tests_targets $target" + fi done -export target_list source_path use_containers cpu host_cc -$source_path/tests/tcg/configure.sh) +echo "TCG_TESTS_TARGETS=$tcg_tests_targets" >> $makefile) config_mak=pc-bios/optionrom/config.mak echo "# Automatically generated by configure - do not modify" > $config_mak diff --git a/tests/Makefile.include b/tests/Makefile.include index 72ce0561f4..6a1688e33e 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -37,7 +37,6 @@ export SRC_PATH SPEED = quick -include tests/tcg/Makefile.prereqs -config-host.mak: $(SRC_PATH)/tests/tcg/configure.sh tests/tcg/Makefile.prereqs: config-host.mak # Per guest TCG tests diff --git a/tests/tcg/configure.sh b/tests/tcg/configure.sh deleted file mode 100755 index 59f2403d1a..0000000000 --- a/tests/tcg/configure.sh +++ /dev/null @@ -1,376 +0,0 @@ -#! /bin/sh - -if test -z "$source_path"; then - echo Do not invoke this script directly. It is called - echo automatically by configure. - exit 1 -fi - -write_c_skeleton() { - cat > $TMPC </dev/null 2>&1 -} - -do_compiler() { - # Run the compiler, capturing its output to the log. First argument - # is compiler binary to execute. - local compiler="$1" - shift - if test -n "$BASH_VERSION"; then eval ' - echo >>config.log " -funcs: ${FUNCNAME[*]} -lines: ${BASH_LINENO[*]}" - '; fi - echo $compiler "$@" >> config.log - $compiler "$@" >> config.log 2>&1 || return $? -} - - -TMPDIR1="config-temp" -TMPC="${TMPDIR1}/qemu-conf.c" -TMPE="${TMPDIR1}/qemu-conf.exe" - -container="no" -if test $use_containers = "yes"; then - if has "docker" || has "podman"; then - container=$($python $source_path/tests/docker/docker.py probe) - fi -fi - -# cross compilers defaults, can be overridden with --cross-cc-ARCH -: ${cross_cc_aarch64="aarch64-linux-gnu-gcc"} -: ${cross_cc_aarch64_be="$cross_cc_aarch64"} -: ${cross_cc_cflags_aarch64_be="-mbig-endian"} -: ${cross_cc_alpha="alpha-linux-gnu-gcc"} -: ${cross_cc_arm="arm-linux-gnueabihf-gcc"} -: ${cross_cc_cflags_armeb="-mbig-endian"} -: ${cross_cc_hexagon="hexagon-unknown-linux-musl-clang"} -: ${cross_cc_cflags_hexagon="-mv67 -O2 -static"} -: ${cross_cc_hppa="hppa-linux-gnu-gcc"} -: ${cross_cc_i386="i686-linux-gnu-gcc"} -: ${cross_cc_cflags_i386="-m32"} -: ${cross_cc_m68k="m68k-linux-gnu-gcc"} -: ${cross_cc_microblaze="microblaze-linux-musl-gcc"} -: ${cross_cc_mips64el="mips64el-linux-gnuabi64-gcc"} -: ${cross_cc_mips64="mips64-linux-gnuabi64-gcc"} -: ${cross_cc_mipsel="mipsel-linux-gnu-gcc"} -: ${cross_cc_mips="mips-linux-gnu-gcc"} -: ${cross_cc_nios2="nios2-linux-gnu-gcc"} -: ${cross_cc_ppc="powerpc-linux-gnu-gcc"} -: ${cross_cc_cflags_ppc="-m32"} -: ${cross_cc_ppc64="powerpc64-linux-gnu-gcc"} -: ${cross_cc_cflags_ppc64="-m64 -mbig-endian"} -: ${cross_cc_ppc64le="$cross_cc_ppc64"} -: ${cross_cc_cflags_ppc64le="-m64 -mlittle-endian"} -: ${cross_cc_riscv64="riscv64-linux-gnu-gcc"} -: ${cross_cc_s390x="s390x-linux-gnu-gcc"} -: ${cross_cc_sh4="sh4-linux-gnu-gcc"} -: ${cross_cc_cflags_sparc="-m32 -mcpu=supersparc"} -: ${cross_cc_sparc64="sparc64-linux-gnu-gcc"} -: ${cross_cc_cflags_sparc64="-m64 -mcpu=ultrasparc"} -: ${cross_cc_x86_64="x86_64-linux-gnu-gcc"} -: ${cross_cc_cflags_x86_64="-m64"} - -# tricore is special as it doesn't have a compiler -: ${cross_as_tricore="tricore-as"} -: ${cross_ld_tricore="tricore-ld"} - -makefile=tests/tcg/Makefile.prereqs -echo "# Automatically generated by configure - do not modify" > $makefile - -config_host_mak=tests/tcg/config-host.mak -echo "# Automatically generated by configure - do not modify" > $config_host_mak -echo "SRC_PATH=$source_path" >> $config_host_mak -echo "HOST_CC=$host_cc" >> $config_host_mak - -tcg_tests_targets= -for target in $target_list; do - arch=${target%%-*} - - # reset all container fields - container_image= - container_hosts= - container_cross_cc= - container_cross_as= - container_cross_ld= - - # suppress clang - supress_clang= - - case $target in - aarch64-*) - # We don't have any bigendian build tools so we only use this for AArch64 - container_hosts="x86_64 aarch64" - container_image=debian-arm64-cross - container_cross_cc=aarch64-linux-gnu-gcc-10 - ;; - alpha-*) - container_hosts=x86_64 - container_image=debian-alpha-cross - container_cross_cc=alpha-linux-gnu-gcc - ;; - arm-*) - # We don't have any bigendian build tools so we only use this for ARM - container_hosts="x86_64 aarch64" - container_image=debian-armhf-cross - container_cross_cc=arm-linux-gnueabihf-gcc - ;; - cris-*) - container_hosts=x86_64 - container_image=fedora-cris-cross - container_cross_cc=cris-linux-gnu-gcc - ;; - hexagon-*) - container_hosts=x86_64 - container_image=debian-hexagon-cross - container_cross_cc=hexagon-unknown-linux-musl-clang - ;; - hppa-*) - container_hosts=x86_64 - container_image=debian-hppa-cross - container_cross_cc=hppa-linux-gnu-gcc - ;; - i386-*) - container_hosts=x86_64 - container_image=fedora-i386-cross - container_cross_cc=gcc - supress_clang=yes - ;; - m68k-*) - container_hosts=x86_64 - container_image=debian-m68k-cross - container_cross_cc=m68k-linux-gnu-gcc - ;; - microblaze-*) - container_hosts=x86_64 - container_image=debian-microblaze-cross - container_cross_cc=microblaze-linux-musl-gcc - ;; - mips64el-*) - container_hosts=x86_64 - container_image=debian-mips64el-cross - container_cross_cc=mips64el-linux-gnuabi64-gcc - ;; - mips64-*) - container_hosts=x86_64 - container_image=debian-mips64-cross - container_cross_cc=mips64-linux-gnuabi64-gcc - ;; - mipsel-*) - container_hosts=x86_64 - container_image=debian-mipsel-cross - container_cross_cc=mipsel-linux-gnu-gcc - ;; - mips-*) - container_hosts=x86_64 - container_image=debian-mips-cross - container_cross_cc=mips-linux-gnu-gcc - ;; - nios2-*) - container_hosts=x86_64 - container_image=debian-nios2-cross - container_cross_cc=nios2-linux-gnu-gcc - ;; - ppc-*) - container_hosts=x86_64 - container_image=debian-powerpc-test-cross - container_cross_cc=powerpc-linux-gnu-gcc-10 - ;; - ppc64-*|ppc64le-*) - container_hosts=x86_64 - container_image=debian-powerpc-test-cross - container_cross_cc=${target%%-*}-linux-gnu-gcc-10 - container_cross_cc=powerpc${container_cross_cc#ppc} - ;; - riscv64-*) - container_hosts=x86_64 - container_image=debian-riscv64-test-cross - container_cross_cc=riscv64-linux-gnu-gcc - ;; - s390x-*) - container_hosts=x86_64 - container_image=debian-s390x-cross - container_cross_cc=s390x-linux-gnu-gcc - ;; - sh4-*) - container_hosts=x86_64 - container_image=debian-sh4-cross - container_cross_cc=sh4-linux-gnu-gcc - ;; - sparc64-*) - container_hosts=x86_64 - container_image=debian-sparc64-cross - container_cross_cc=sparc64-linux-gnu-gcc - ;; - tricore-softmmu) - container_hosts=x86_64 - container_image=debian-tricore-cross - container_cross_as=tricore-as - container_cross_ld=tricore-ld - ;; - x86_64-*) - container_hosts="aarch64 ppc64el x86_64" - container_image=debian-amd64-cross - container_cross_cc=x86_64-linux-gnu-gcc - supress_clang=yes - ;; - xtensa*-softmmu) - container_hosts=x86_64 - container_image=debian-xtensa-cross - - # default to the dc232b cpu - container_cross_cc=/opt/2020.07/xtensa-dc232b-elf/bin/xtensa-dc232b-elf-gcc - ;; - esac - - config_target_mak=tests/tcg/config-$target.mak - - echo "# Automatically generated by configure - do not modify" > $config_target_mak - echo "TARGET_NAME=$arch" >> $config_target_mak - case $target in - *-softmmu) - test -f $source_path/tests/tcg/$arch/Makefile.softmmu-target || continue - qemu="qemu-system-$arch" - ;; - *-linux-user|*-bsd-user) - qemu="qemu-$arch" - ;; - esac - - eval "target_compiler_cflags=\${cross_cc_cflags_$arch}" - - got_cross_cc=no - - if eval test "x\"\${cross_cc_$arch}\"" != xyes; then - eval "target_compiler=\"\${cross_cc_$arch}\"" - - if has $target_compiler; then - if test "$supress_clang" = yes && - $target_compiler --version | grep -qi "clang"; then - got_cross_cc=no - else - write_c_skeleton - if ! do_compiler "$target_compiler" $target_compiler_cflags \ - -o $TMPE $TMPC -static ; then - # For host systems we might get away with building without -static - if do_compiler "$target_compiler" $target_compiler_cflags \ - -o $TMPE $TMPC ; then - got_cross_cc=yes - echo "CC=$target_compiler" >> $config_target_mak - fi - else - got_cross_cc=yes - echo "BUILD_STATIC=y" >> $config_target_mak - echo "CC=$target_compiler" >> $config_target_mak - fi - fi - fi - - # Special handling for assembler only tests - eval "target_as=\"\${cross_as_$arch}\"" - eval "target_ld=\"\${cross_ld_$arch}\"" - if has $target_as && has $target_ld; then - case $target in - tricore-softmmu) - echo "AS=$target_as" >> $config_target_mak - echo "LD=$target_ld" >> $config_target_mak - got_cross_cc=yes - ;; - esac - fi - fi - - if test $got_cross_cc = yes; then - # Test for compiler features for optional tests. We only do this - # for cross compilers because ensuring the docker containers based - # compilers is a requirememt for adding a new test that needs a - # compiler feature. - - case $target in - aarch64-*) - if do_compiler "$target_compiler" $target_compiler_cflags \ - -march=armv8.1-a+sve -o $TMPE $TMPC; then - echo "CROSS_CC_HAS_SVE=y" >> $config_target_mak - fi - if do_compiler "$target_compiler" $target_compiler_cflags \ - -march=armv8.1-a+sve2 -o $TMPE $TMPC; then - echo "CROSS_CC_HAS_SVE2=y" >> $config_target_mak - fi - if do_compiler "$target_compiler" $target_compiler_cflags \ - -march=armv8.3-a -o $TMPE $TMPC; then - echo "CROSS_CC_HAS_ARMV8_3=y" >> $config_target_mak - fi - if do_compiler "$target_compiler" $target_compiler_cflags \ - -mbranch-protection=standard -o $TMPE $TMPC; then - echo "CROSS_CC_HAS_ARMV8_BTI=y" >> $config_target_mak - fi - if do_compiler "$target_compiler" $target_compiler_cflags \ - -march=armv8.5-a+memtag -o $TMPE $TMPC; then - echo "CROSS_CC_HAS_ARMV8_MTE=y" >> $config_target_mak - fi - ;; - ppc*) - if do_compiler "$target_compiler" $target_compiler_cflags \ - -mpower8-vector -o $TMPE $TMPC; then - echo "CROSS_CC_HAS_POWER8_VECTOR=y" >> $config_target_mak - fi - if do_compiler "$target_compiler" $target_compiler_cflags \ - -mpower10 -o $TMPE $TMPC; then - echo "CROSS_CC_HAS_POWER10=y" >> $config_target_mak - fi - ;; - i386-linux-user) - if do_compiler "$target_compiler" $target_compiler_cflags \ - -Werror -fno-pie -o $TMPE $TMPC; then - echo "CROSS_CC_HAS_I386_NOPIE=y" >> $config_target_mak - fi - ;; - esac - elif test $got_cross_cc = no && test "$container" != no && \ - test -n "$container_image"; then - for host in $container_hosts; do - if test "$host" = "$cpu"; then - echo "build-tcg-tests-$target: docker-image-$container_image" >> $makefile - echo "BUILD_STATIC=y" >> $config_target_mak - echo "CC=\$(DOCKER_SCRIPT) cc --cc $container_cross_cc -i qemu/$container_image -s $source_path --" >> $config_target_mak - if test -n "$container_cross_as"; then - echo "AS=\$(DOCKER_SCRIPT) cc --cc $container_cross_as -i qemu/$container_image -s $source_path --" >> $config_target_mak - fi - if test -n "$container_cross_ld"; then - echo "LD=\$(DOCKER_SCRIPT) cc --cc $container_cross_ld -i qemu/$container_image -s $source_path --" >> $config_target_mak - fi - case $target in - aarch64-*) - echo "CROSS_CC_HAS_SVE=y" >> $config_target_mak - echo "CROSS_CC_HAS_SVE2=y" >> $config_target_mak - echo "CROSS_CC_HAS_ARMV8_3=y" >> $config_target_mak - echo "CROSS_CC_HAS_ARMV8_BTI=y" >> $config_target_mak - echo "CROSS_CC_HAS_ARMV8_MTE=y" >> $config_target_mak - ;; - ppc*) - echo "CROSS_CC_HAS_POWER8_VECTOR=y" >> $config_target_mak - echo "CROSS_CC_HAS_POWER10=y" >> $config_target_mak - ;; - i386-linux-user) - echo "CROSS_CC_HAS_I386_NOPIE=y" >> $config_target_mak - ;; - esac - got_cross_cc=yes - break - fi - done - fi - if test $got_cross_cc = yes; then - mkdir -p tests/tcg/$target - echo "QEMU=$PWD/$qemu" >> $config_target_mak - echo "EXTRA_CFLAGS=$target_compiler_cflags" >> $config_target_mak - echo "run-tcg-tests-$target: $qemu\$(EXESUF)" >> $makefile - tcg_tests_targets="$tcg_tests_targets $target" - fi -done -echo "TCG_TESTS_TARGETS=$tcg_tests_targets" >> $makefile From 46af66edcb9c123069a941f495ba8b13f486851a Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Fri, 27 May 2022 16:35:49 +0100 Subject: [PATCH 19/33] configure: add missing cross compiler fallbacks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The arm compiler can be used for armeb, and the sparc64 compiler can be used for sparc. Signed-off-by: Paolo Bonzini Reviewed-by: Richard Henderson Message-Id: <20220517092616.1272238-9-pbonzini@redhat.com> Signed-off-by: Alex Bennée Message-Id: <20220527153603.887929-20-alex.bennee@linaro.org> --- configure | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/configure b/configure index f91ac632e7..fbf6d39f96 100755 --- a/configure +++ b/configure @@ -1827,6 +1827,7 @@ fi : ${cross_cc_cflags_aarch64_be="-mbig-endian"} : ${cross_cc_alpha="alpha-linux-gnu-gcc"} : ${cross_cc_arm="arm-linux-gnueabihf-gcc"} +: ${cross_cc_armeb="$cross_cc_arm"} : ${cross_cc_cflags_armeb="-mbig-endian"} : ${cross_cc_hexagon="hexagon-unknown-linux-musl-clang"} : ${cross_cc_cflags_hexagon="-mv67 -O2 -static"} @@ -1849,9 +1850,10 @@ fi : ${cross_cc_riscv64="riscv64-linux-gnu-gcc"} : ${cross_cc_s390x="s390x-linux-gnu-gcc"} : ${cross_cc_sh4="sh4-linux-gnu-gcc"} -: ${cross_cc_cflags_sparc="-m32 -mcpu=supersparc"} : ${cross_cc_sparc64="sparc64-linux-gnu-gcc"} : ${cross_cc_cflags_sparc64="-m64 -mcpu=ultrasparc"} +: ${cross_cc_sparc="$cross_cc_sparc64"} +: ${cross_cc_cflags_sparc="-m32 -mcpu=supersparc"} : ${cross_cc_x86_64="x86_64-linux-gnu-gcc"} : ${cross_cc_cflags_x86_64="-m64"} From 2ad60f6f8c12ca0acd8834fdd70e088361b8791f Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Fri, 27 May 2022 16:35:50 +0100 Subject: [PATCH 20/33] configure: handle host compiler in probe_target_compiler MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit In preparation for handling more binaries than just cc, handle the case of "probe_target_compiler $cpu" directly in the function, setting the target_* variables based on the ones that are used to build QEMU. The clang check also needs to be moved after this fallback. Signed-off-by: Paolo Bonzini Reviewed-by: Richard Henderson Message-Id: <20220517092616.1272238-10-pbonzini@redhat.com> Signed-off-by: Alex Bennée Message-Id: <20220527153603.887929-21-alex.bennee@linaro.org> --- configure | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/configure b/configure index fbf6d39f96..217c8b3cac 100755 --- a/configure +++ b/configure @@ -954,10 +954,6 @@ case $git_submodules_action in ;; esac -if eval test -z "\${cross_cc_$cpu}"; then - eval "cross_cc_${cpu}=\$cc" -fi - default_target_list="" mak_wilds="" @@ -2008,13 +2004,6 @@ probe_target_compiler() { if eval test -n "\"\${cross_cc_$1}\""; then if eval has "\"\${cross_cc_$1}\""; then eval "target_cc=\"\${cross_cc_$1}\"" - case $1 in - i386|x86_64) - if $target_cc --version | grep -qi "clang"; then - unset target_cc - fi - ;; - esac fi fi if eval test -n "\"\${cross_as_$1}\""; then @@ -2027,6 +2016,20 @@ probe_target_compiler() { eval "target_ld=\"\${cross_ld_$1}\"" fi fi + if test "$1" = $cpu; then + : ${target_cc:=$cc} + : ${target_as:=$as} + : ${target_ld:=$ld} + fi + if test -n "$target_cc"; then + case $1 in + i386|x86_64) + if $target_cc --version | grep -qi "clang"; then + unset target_cc + fi + ;; + esac + fi } write_target_makefile() { From 5adb43be79dd125f8bf7bce0fd4b0c4f03e0aa71 Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Fri, 27 May 2022 16:35:51 +0100 Subject: [PATCH 21/33] configure: introduce --cross-prefix-*= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Also in preparation for handling more binaries from the cross binutils, support an option --cross-prefix-ARCH. All cross_cc_* defaults are replaced with cross_prefix_*; the cross_cc_* fallbacks are extended to the cross-compilation prefix, but the compiler fallbacks remain as well. This way, for example, --cross-cc-arm=arm-linux-gnueabihf-clang also applies to armeb binaries. Reviewed-by: Richard Henderson Signed-off-by: Paolo Bonzini Message-Id: <20220517092616.1272238-11-pbonzini@redhat.com> Signed-off-by: Alex Bennée Message-Id: <20220527153603.887929-22-alex.bennee@linaro.org> --- configure | 137 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 77 insertions(+), 60 deletions(-) diff --git a/configure b/configure index 217c8b3cac..f55ae82a5d 100755 --- a/configure +++ b/configure @@ -363,6 +363,11 @@ for opt do --cross-cc-*) cc_arch=${opt#--cross-cc-}; cc_arch=${cc_arch%%=*} eval "cross_cc_${cc_arch}=\$optarg" ;; + --cross-prefix-*[!a-zA-Z0-9_-]*=*) error_exit "Passed bad --cross-prefix-FOO option" + ;; + --cross-prefix-*) cc_arch=${opt#--cross-prefix-}; cc_arch=${cc_arch%%=*} + eval "cross_prefix_${cc_arch}=\$optarg" + ;; esac done # OS specific @@ -728,6 +733,8 @@ for opt do ;; --cross-cc-*) ;; + --cross-prefix-*) + ;; --enable-debug-info) meson_option_add -Ddebug=true ;; --disable-debug-info) meson_option_add -Ddebug=false @@ -1016,6 +1023,7 @@ Advanced options (experts only): --extra-ldflags=LDFLAGS append extra linker flags LDFLAGS --cross-cc-ARCH=CC use compiler when building ARCH guest test cases --cross-cc-cflags-ARCH= use compiler flags when building ARCH guest tests + --cross-prefix-ARCH=PREFIX cross compiler prefix when building ARCH guest test cases --make=MAKE use specified make [$make] --python=PYTHON use specified python [$python] --meson=MESON use specified meson [$meson] @@ -1818,44 +1826,54 @@ if test $use_containers = "yes"; then fi # cross compilers defaults, can be overridden with --cross-cc-ARCH -: ${cross_cc_aarch64="aarch64-linux-gnu-gcc"} +: ${cross_prefix_aarch64="aarch64-linux-gnu-"} +: ${cross_prefix_aarch64_be="$cross_prefix_aarch64"} +: ${cross_prefix_alpha="alpha-linux-gnu-"} +: ${cross_prefix_arm="arm-linux-gnueabihf-"} +: ${cross_prefix_armeb="$cross_prefix_arm"} +: ${cross_prefix_hexagon="hexagon-unknown-linux-musl-"} +: ${cross_prefix_hppa="hppa-linux-gnu-"} +: ${cross_prefix_i386="i686-linux-gnu-"} +: ${cross_prefix_m68k="m68k-linux-gnu-"} +: ${cross_prefix_microblaze="microblaze-linux-musl-"} +: ${cross_prefix_mips64el="mips64el-linux-gnuabi64-"} +: ${cross_prefix_mips64="mips64-linux-gnuabi64-"} +: ${cross_prefix_mipsel="mipsel-linux-gnu-"} +: ${cross_prefix_mips="mips-linux-gnu-"} +: ${cross_prefix_nios2="nios2-linux-gnu-"} +: ${cross_prefix_ppc="powerpc-linux-gnu-"} +: ${cross_prefix_ppc64="powerpc64-linux-gnu-"} +: ${cross_prefix_ppc64le="$cross_prefix_ppc64"} +: ${cross_prefix_riscv64="riscv64-linux-gnu-"} +: ${cross_prefix_s390x="s390x-linux-gnu-"} +: ${cross_prefix_sh4="sh4-linux-gnu-"} +: ${cross_prefix_sparc64="sparc64-linux-gnu-"} +: ${cross_prefix_sparc="$cross_prefix_sparc64"} +: ${cross_prefix_x86_64="x86_64-linux-gnu-"} + : ${cross_cc_aarch64_be="$cross_cc_aarch64"} : ${cross_cc_cflags_aarch64_be="-mbig-endian"} -: ${cross_cc_alpha="alpha-linux-gnu-gcc"} -: ${cross_cc_arm="arm-linux-gnueabihf-gcc"} : ${cross_cc_armeb="$cross_cc_arm"} : ${cross_cc_cflags_armeb="-mbig-endian"} : ${cross_cc_hexagon="hexagon-unknown-linux-musl-clang"} : ${cross_cc_cflags_hexagon="-mv67 -O2 -static"} -: ${cross_cc_hppa="hppa-linux-gnu-gcc"} -: ${cross_cc_i386="i686-linux-gnu-gcc"} : ${cross_cc_cflags_i386="-m32"} -: ${cross_cc_m68k="m68k-linux-gnu-gcc"} -: ${cross_cc_microblaze="microblaze-linux-musl-gcc"} -: ${cross_cc_mips64el="mips64el-linux-gnuabi64-gcc"} -: ${cross_cc_mips64="mips64-linux-gnuabi64-gcc"} -: ${cross_cc_mipsel="mipsel-linux-gnu-gcc"} -: ${cross_cc_mips="mips-linux-gnu-gcc"} -: ${cross_cc_nios2="nios2-linux-gnu-gcc"} -: ${cross_cc_ppc="powerpc-linux-gnu-gcc"} : ${cross_cc_cflags_ppc="-m32"} -: ${cross_cc_ppc64="powerpc64-linux-gnu-gcc"} : ${cross_cc_cflags_ppc64="-m64 -mbig-endian"} : ${cross_cc_ppc64le="$cross_cc_ppc64"} : ${cross_cc_cflags_ppc64le="-m64 -mlittle-endian"} -: ${cross_cc_riscv64="riscv64-linux-gnu-gcc"} -: ${cross_cc_s390x="s390x-linux-gnu-gcc"} -: ${cross_cc_sh4="sh4-linux-gnu-gcc"} -: ${cross_cc_sparc64="sparc64-linux-gnu-gcc"} : ${cross_cc_cflags_sparc64="-m64 -mcpu=ultrasparc"} : ${cross_cc_sparc="$cross_cc_sparc64"} : ${cross_cc_cflags_sparc="-m32 -mcpu=supersparc"} -: ${cross_cc_x86_64="x86_64-linux-gnu-gcc"} : ${cross_cc_cflags_x86_64="-m64"} -# tricore is special as it doesn't have a compiler -: ${cross_as_tricore="tricore-as"} -: ${cross_ld_tricore="tricore-ld"} +compute_target_variable() { + if eval test -n "\"\${cross_prefix_$1}\""; then + if eval has "\"\${cross_prefix_$1}\$3\""; then + eval "$2=\"\${cross_prefix_$1}\$3\"" + fi + fi +} probe_target_compiler() { # reset all output variables @@ -1901,93 +1919,99 @@ probe_target_compiler() { aarch64) # We don't have any bigendian build tools so we only use this for AArch64 container_image=debian-arm64-cross - container_cross_cc=aarch64-linux-gnu-gcc-10 + container_cross_prefix=aarch64-linux-gnu- + container_cross_cc=${container_cross_prefix}gcc-10 ;; alpha) container_image=debian-alpha-cross - container_cross_cc=alpha-linux-gnu-gcc + container_cross_prefix=alpha-linux-gnu- ;; arm) # We don't have any bigendian build tools so we only use this for ARM container_image=debian-armhf-cross - container_cross_cc=arm-linux-gnueabihf-gcc + container_cross_prefix=arm-linux-gnueabihf- ;; cris) container_image=fedora-cris-cross - container_cross_cc=cris-linux-gnu-gcc + container_cross_prefix=cris-linux-gnu- ;; hexagon) container_image=debian-hexagon-cross - container_cross_cc=hexagon-unknown-linux-musl-clang + container_cross_prefix=hexagon-unknown-linux-musl- + container_cross_cc=${container_cross_prefix}clang ;; hppa) container_image=debian-hppa-cross - container_cross_cc=hppa-linux-gnu-gcc + container_cross_prefix=hppa-linux-gnu- ;; i386) container_image=fedora-i386-cross - container_cross_cc=gcc + container_cross_prefix= ;; m68k) container_image=debian-m68k-cross - container_cross_cc=m68k-linux-gnu-gcc + container_cross_prefix=m68k-linux-gnu- ;; microblaze) container_image=debian-microblaze-cross - container_cross_cc=microblaze-linux-musl-gcc + container_cross_prefix=microblaze-linux-musl- ;; mips64el) container_image=debian-mips64el-cross - container_cross_cc=mips64el-linux-gnuabi64-gcc + container_cross_prefix=mips64el-linux-gnuabi64- ;; mips64) container_image=debian-mips64-cross - container_cross_cc=mips64-linux-gnuabi64-gcc + container_cross_prefix=mips64-linux-gnuabi64- ;; mipsel) container_image=debian-mipsel-cross - container_cross_cc=mipsel-linux-gnu-gcc + container_cross_prefix=mipsel-linux-gnu- ;; mips) container_image=debian-mips-cross - container_cross_cc=mips-linux-gnu-gcc + container_cross_prefix=mips-linux-gnu- ;; nios2) container_image=debian-nios2-cross - container_cross_cc=nios2-linux-gnu-gcc + container_cross_prefix=nios2-linux-gnu- ;; ppc) container_image=debian-powerpc-test-cross - container_cross_cc=powerpc-linux-gnu-gcc-10 + container_cross_prefix=powerpc-linux-gnu- + container_cross_cc=${container_cross_prefix}gcc-10 ;; ppc64|ppc64le) container_image=debian-powerpc-test-cross - container_cross_cc=powerpc${1#ppc}-linux-gnu-gcc-10 + container_cross_prefix=powerpc${1#ppc}-linux-gnu- + container_cross_cc=${container_cross_prefix}gcc-10 ;; riscv64) container_image=debian-riscv64-test-cross - container_cross_cc=riscv64-linux-gnu-gcc + container_cross_prefix=riscv64-linux-gnu- ;; s390x) container_image=debian-s390x-cross - container_cross_cc=s390x-linux-gnu-gcc + container_cross_prefix=s390x-linux-gnu- ;; sh4) container_image=debian-sh4-cross - container_cross_cc=sh4-linux-gnu-gcc + container_cross_prefix=sh4-linux-gnu- ;; sparc64) container_image=debian-sparc64-cross - container_cross_cc=sparc64-linux-gnu-gcc + container_cross_prefix=sparc64-linux-gnu- ;; tricore) container_image=debian-tricore-cross + container_cross_prefix=tricore- container_cross_as=tricore-as container_cross_ld=tricore-ld + break ;; x86_64) container_image=debian-amd64-cross - container_cross_cc=x86_64-linux-gnu-gcc + container_cross_prefix=x86_64-linux-gnu- ;; xtensa*) # FIXME: xtensa-linux-user? @@ -1995,9 +2019,12 @@ probe_target_compiler() { container_image=debian-xtensa-cross # default to the dc232b cpu - container_cross_cc=/opt/2020.07/xtensa-dc232b-elf/bin/xtensa-dc232b-elf-gcc + container_cross_prefix=/opt/2020.07/xtensa-dc232b-elf/bin/xtensa-dc232b-elf- ;; esac + : ${container_cross_cc:=${container_cross_prefix}gcc} + : ${container_cross_as:=${container_cross_prefix}as} + : ${container_cross_ld:=${container_cross_prefix}ld} done eval "target_cflags=\${cross_cc_cflags_$1}" @@ -2005,17 +2032,11 @@ probe_target_compiler() { if eval has "\"\${cross_cc_$1}\""; then eval "target_cc=\"\${cross_cc_$1}\"" fi + else + compute_target_variable $1 target_cc gcc fi - if eval test -n "\"\${cross_as_$1}\""; then - if eval has "\"\${cross_as_$1}\""; then - eval "target_as=\"\${cross_as_$1}\"" - fi - fi - if eval test -n "\"\${cross_ld_$1}\""; then - if eval has "\"\${cross_ld_$1}\""; then - eval "target_ld=\"\${cross_ld_$1}\"" - fi - fi + compute_target_variable $1 target_as as + compute_target_variable $1 target_ld ld if test "$1" = $cpu; then : ${target_cc:=$cc} : ${target_as:=$as} @@ -2048,12 +2069,8 @@ write_container_target_makefile() { if test -n "$container_cross_cc"; then echo "CC=\$(DOCKER_SCRIPT) cc --cc $container_cross_cc -i qemu/$container_image -s $source_path --" fi - if test -n "$container_cross_as"; then - echo "AS=\$(DOCKER_SCRIPT) cc --cc $container_cross_as -i qemu/$container_image -s $source_path --" - fi - if test -n "$container_cross_ld"; then - echo "LD=\$(DOCKER_SCRIPT) cc --cc $container_cross_ld -i qemu/$container_image -s $source_path --" - fi + echo "AS=\$(DOCKER_SCRIPT) cc --cc $container_cross_as -i qemu/$container_image -s $source_path --" + echo "LD=\$(DOCKER_SCRIPT) cc --cc $container_cross_ld -i qemu/$container_image -s $source_path --" } From 87eb014c5e80039bf6bee4f05557f48827239cb4 Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Fri, 27 May 2022 16:35:52 +0100 Subject: [PATCH 22/33] configure: include more binutils in tests/tcg makefile MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Firmware builds require paths to all the binutils; it is not enough to use only cc, or even as/ld as in the case of tests/tcg/tricore. Adjust the cross-compiler configurator to detect also ar, nm, objcopy, ranlib and strip. Reviewed-by: Richard Henderson Signed-off-by: Paolo Bonzini Message-Id: <20220517092616.1272238-12-pbonzini@redhat.com> Signed-off-by: Alex Bennée Message-Id: <20220527153603.887929-23-alex.bennee@linaro.org> --- configure | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/configure b/configure index f55ae82a5d..ced8283b98 100755 --- a/configure +++ b/configure @@ -1880,11 +1880,21 @@ probe_target_compiler() { container_image= container_hosts= container_cross_cc= + container_cross_ar= container_cross_as= container_cross_ld= + container_cross_nm= + container_cross_objcopy= + container_cross_ranlib= + container_cross_strip= target_cc= + target_ar= target_as= target_ld= + target_nm= + target_objcopy= + target_ranlib= + target_strip= case $1 in aarch64) container_hosts="x86_64 aarch64" ;; @@ -2023,8 +2033,13 @@ probe_target_compiler() { ;; esac : ${container_cross_cc:=${container_cross_prefix}gcc} + : ${container_cross_ar:=${container_cross_prefix}ar} : ${container_cross_as:=${container_cross_prefix}as} : ${container_cross_ld:=${container_cross_prefix}ld} + : ${container_cross_nm:=${container_cross_prefix}nm} + : ${container_cross_objcopy:=${container_cross_prefix}objcopy} + : ${container_cross_ranlib:=${container_cross_prefix}ranlib} + : ${container_cross_strip:=${container_cross_prefix}strip} done eval "target_cflags=\${cross_cc_cflags_$1}" @@ -2035,12 +2050,26 @@ probe_target_compiler() { else compute_target_variable $1 target_cc gcc fi + target_ccas=$target_cc + compute_target_variable $1 target_ar ar compute_target_variable $1 target_as as compute_target_variable $1 target_ld ld + compute_target_variable $1 target_nm nm + compute_target_variable $1 target_objcopy objcopy + compute_target_variable $1 target_ranlib ranlib + compute_target_variable $1 target_strip strip if test "$1" = $cpu; then : ${target_cc:=$cc} + : ${target_ccas:=$ccas} : ${target_as:=$as} : ${target_ld:=$ld} + : ${target_ar:=$ar} + : ${target_as:=$as} + : ${target_ld:=$ld} + : ${target_nm:=$nm} + : ${target_objcopy:=$objcopy} + : ${target_ranlib:=$ranlib} + : ${target_strip:=$strip} fi if test -n "$target_cc"; then case $1 in @@ -2056,6 +2085,10 @@ probe_target_compiler() { write_target_makefile() { if test -n "$target_cc"; then echo "CC=$target_cc" + echo "CCAS=$target_ccas" + fi + if test -n "$target_ar"; then + echo "AR=$target_ar" fi if test -n "$target_as"; then echo "AS=$target_as" @@ -2063,14 +2096,32 @@ write_target_makefile() { if test -n "$target_ld"; then echo "LD=$target_ld" fi + if test -n "$target_nm"; then + echo "NM=$target_nm" + fi + if test -n "$target_objcopy"; then + echo "OBJCOPY=$target_objcopy" + fi + if test -n "$target_ranlib"; then + echo "RANLIB=$target_ranlib" + fi + if test -n "$target_strip"; then + echo "STRIP=$target_strip" + fi } write_container_target_makefile() { if test -n "$container_cross_cc"; then echo "CC=\$(DOCKER_SCRIPT) cc --cc $container_cross_cc -i qemu/$container_image -s $source_path --" + echo "CCAS=\$(DOCKER_SCRIPT) cc --cc $container_cross_cc -i qemu/$container_image -s $source_path --" fi + echo "AR=\$(DOCKER_SCRIPT) cc --cc $container_cross_ar -i qemu/$container_image -s $source_path --" echo "AS=\$(DOCKER_SCRIPT) cc --cc $container_cross_as -i qemu/$container_image -s $source_path --" echo "LD=\$(DOCKER_SCRIPT) cc --cc $container_cross_ld -i qemu/$container_image -s $source_path --" + echo "NM=\$(DOCKER_SCRIPT) cc --cc $container_cross_nm -i qemu/$container_image -s $source_path --" + echo "OBJCOPY=\$(DOCKER_SCRIPT) cc --cc $container_cross_objcopy -i qemu/$container_image -s $source_path --" + echo "RANLIB=\$(DOCKER_SCRIPT) cc --cc $container_cross_ranlib -i qemu/$container_image -s $source_path --" + echo "STRIP=\$(DOCKER_SCRIPT) cc --cc $container_cross_strip -i qemu/$container_image -s $source_path --" } From ad38884512a72206463a95a30451253946d4218b Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Fri, 27 May 2022 16:35:53 +0100 Subject: [PATCH 23/33] configure: move symlink configuration earlier MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ensure that the pc-bios/optionrom and pc-bios/s390-ccw directory exist at the time when we'll write out the compiler configuration for them. Reviewed-by: Richard Henderson Signed-off-by: Paolo Bonzini Message-Id: <20220517092616.1272238-13-pbonzini@redhat.com> Signed-off-by: Alex Bennée Message-Id: <20220527153603.887929-24-alex.bennee@linaro.org> --- configure | 49 ++++++++++++++++++++++++------------------------- 1 file changed, 24 insertions(+), 25 deletions(-) diff --git a/configure b/configure index ced8283b98..aa17b0fa55 100755 --- a/configure +++ b/configure @@ -2190,6 +2190,30 @@ if test "$QEMU_GA_VERSION" = ""; then QEMU_GA_VERSION=$(cat $source_path/VERSION) fi +# Set up build tree symlinks that point back into the source tree +# (these can be both files and directories). +# Caution: avoid adding files or directories here using wildcards. This +# will result in problems later if a new file matching the wildcard is +# added to the source tree -- nothing will cause configure to be rerun +# so the build tree will be missing the link back to the new file, and +# tests might fail. Prefer to keep the relevant files in their own +# directory and symlink the directory instead. +LINKS="Makefile" +LINKS="$LINKS tests/tcg/Makefile.target" +LINKS="$LINKS pc-bios/optionrom/Makefile" +LINKS="$LINKS pc-bios/s390-ccw/Makefile" +LINKS="$LINKS .gdbinit scripts" # scripts needed by relative path in .gdbinit +LINKS="$LINKS tests/avocado tests/data" +LINKS="$LINKS tests/qemu-iotests/check" +LINKS="$LINKS python" +LINKS="$LINKS contrib/plugins/Makefile " +for f in $LINKS ; do + if [ -e "$source_path/$f" ]; then + mkdir -p `dirname ./$f` + symlink "$source_path/$f" "$f" + fi +done + # Mac OS X ships with a broken assembler roms= if { test "$cpu" = "i386" || test "$cpu" = "x86_64"; } && \ @@ -2408,31 +2432,6 @@ if test "$safe_stack" = "yes"; then echo "CONFIG_SAFESTACK=y" >> $config_host_mak fi -# If we're using a separate build tree, set it up now. -# LINKS are things to symlink back into the source tree -# (these can be both files and directories). -# Caution: do not add files or directories here using wildcards. This -# will result in problems later if a new file matching the wildcard is -# added to the source tree -- nothing will cause configure to be rerun -# so the build tree will be missing the link back to the new file, and -# tests might fail. Prefer to keep the relevant files in their own -# directory and symlink the directory instead. -LINKS="Makefile" -LINKS="$LINKS tests/tcg/Makefile.target" -LINKS="$LINKS pc-bios/optionrom/Makefile" -LINKS="$LINKS pc-bios/s390-ccw/Makefile" -LINKS="$LINKS .gdbinit scripts" # scripts needed by relative path in .gdbinit -LINKS="$LINKS tests/avocado tests/data" -LINKS="$LINKS tests/qemu-iotests/check" -LINKS="$LINKS python" -LINKS="$LINKS contrib/plugins/Makefile " -for f in $LINKS ; do - if [ -e "$source_path/$f" ]; then - mkdir -p `dirname ./$f` - symlink "$source_path/$f" "$f" - fi -done - # tests/tcg configuration (makefile=tests/tcg/Makefile.prereqs echo "# Automatically generated by configure - do not modify" > $makefile From 9ffed42614585026834fd63cdf586b78b1f72c66 Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Fri, 27 May 2022 16:35:54 +0100 Subject: [PATCH 24/33] configure: enable cross-compilation of s390-ccw MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit While container-based cross compilers are not supported, this already makes it possible to build s390-ccw on any machine that has s390x GCC and binutils installed. Reviewed-by: Richard Henderson Signed-off-by: Paolo Bonzini Message-Id: <20220517092616.1272238-14-pbonzini@redhat.com> Signed-off-by: Alex Bennée Acked-by: Thomas Huth Message-Id: <20220527153603.887929-25-alex.bennee@linaro.org> --- configure | 18 +++++++++++++----- pc-bios/s390-ccw/Makefile | 9 +++++---- pc-bios/s390-ccw/netboot.mak | 2 +- 3 files changed, 19 insertions(+), 10 deletions(-) diff --git a/configure b/configure index aa17b0fa55..f509bce304 100755 --- a/configure +++ b/configure @@ -2231,24 +2231,32 @@ if { test "$cpu" = "i386" || test "$cpu" = "x86_64"; } && \ done fi -# Only build s390-ccw bios if we're on s390x and the compiler has -march=z900 -# or -march=z10 (which is the lowest architecture level that Clang supports) -if test "$cpu" = "s390x" ; then +# Only build s390-ccw bios if the compiler has -march=z900 or -march=z10 +# (which is the lowest architecture level that Clang supports) +probe_target_compiler s390x +if test -n "$target_cc" && test "$softmmu" = yes; then write_c_skeleton - compile_prog "-march=z900" "" + do_compiler "$target_cc" $target_cc_cflags -march=z900 -o $TMPO -c $TMPC has_z900=$? - if [ $has_z900 = 0 ] || compile_object "-march=z10 -msoft-float -Werror"; then + if [ $has_z900 = 0 ] || do_compiler "$target_cc" $target_cc_cflags -march=z10 -msoft-float -Werror -o $TMPO -c $TMPC; then if [ $has_z900 != 0 ]; then echo "WARNING: Your compiler does not support the z900!" echo " The s390-ccw bios will only work with guest CPUs >= z10." fi roms="$roms s390-ccw" + config_mak=pc-bios/s390-ccw/config-host.mak + echo "# Automatically generated by configure - do not modify" > $config_mak + echo "SRC_PATH=$source_path/pc-bios/s390-ccw" >> $config_mak + write_target_makefile >> $config_mak # SLOF is required for building the s390-ccw firmware on s390x, # since it is using the libnet code from SLOF for network booting. git_submodules="${git_submodules} roms/SLOF" fi fi +####################################### +# generate config-host.mak + # Check that the C++ compiler exists and works with the C compiler. # All the QEMU_CXXFLAGS are based on QEMU_CFLAGS. Keep this at the end to don't miss any other that could be added. if has $cxx; then diff --git a/pc-bios/s390-ccw/Makefile b/pc-bios/s390-ccw/Makefile index 0eb68efc7b..6eb713bf37 100644 --- a/pc-bios/s390-ccw/Makefile +++ b/pc-bios/s390-ccw/Makefile @@ -2,8 +2,9 @@ all: build-all # Dummy command so that make thinks it has done something @true -include ../../config-host.mak +include config-host.mak CFLAGS = -O2 -g +MAKEFLAGS += -rR quiet-command = $(if $(V),$1,$(if $(2),@printf " %-7s %s\n" $2 $3 && $1, @$1)) cc-option = $(if $(shell $(CC) $1 $2 -S -o /dev/null -xc /dev/null \ @@ -11,7 +12,7 @@ cc-option = $(if $(shell $(CC) $1 $2 -S -o /dev/null -xc /dev/null \ VPATH_SUFFIXES = %.c %.h %.S %.m %.mak %.sh %.rc Kconfig% %.json.in set-vpath = $(if $1,$(foreach PATTERN,$(VPATH_SUFFIXES),$(eval vpath $(PATTERN) $1))) -$(call set-vpath, $(SRC_PATH)/pc-bios/s390-ccw) +$(call set-vpath, $(SRC_PATH)) # Flags for dependency generation QEMU_DGFLAGS = -MMD -MP -MT $@ -MF $(@D)/$(*F).d @@ -49,8 +50,8 @@ s390-ccw.img: s390-ccw.elf $(OBJECTS): Makefile -ifneq ($(wildcard $(SRC_PATH)/roms/SLOF/lib/libnet),) -include $(SRC_PATH)/pc-bios/s390-ccw/netboot.mak +ifneq ($(wildcard $(SRC_PATH)/../../roms/SLOF/lib/libnet),) +include $(SRC_PATH)/netboot.mak else s390-netboot.img: @echo "s390-netboot.img not built since roms/SLOF/ is not available." diff --git a/pc-bios/s390-ccw/netboot.mak b/pc-bios/s390-ccw/netboot.mak index 68b4d7edcb..1a06befa4b 100644 --- a/pc-bios/s390-ccw/netboot.mak +++ b/pc-bios/s390-ccw/netboot.mak @@ -1,5 +1,5 @@ -SLOF_DIR := $(SRC_PATH)/roms/SLOF +SLOF_DIR := $(SRC_PATH)/../../roms/SLOF NETOBJS := start.o sclp.o cio.o virtio.o virtio-net.o jump2ipl.o netmain.o From 33ab4787422b04a3f4c94bbded6b05021d08b27d Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Fri, 27 May 2022 16:35:55 +0100 Subject: [PATCH 25/33] configure: enable cross-compilation of optionrom MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit While container-based cross compilers are not supported, this already makes it possible to build x86 optionroms on any machine that has an installation of GCC and binutils for 32- or 64-bit x86. Reviewed-by: Richard Henderson Signed-off-by: Paolo Bonzini Message-Id: <20220517092616.1272238-15-pbonzini@redhat.com> Signed-off-by: Alex Bennée Message-Id: <20220527153603.887929-26-alex.bennee@linaro.org> --- configure | 30 ++++++++++++++++++++++-------- pc-bios/optionrom/Makefile | 2 -- 2 files changed, 22 insertions(+), 10 deletions(-) diff --git a/configure b/configure index f509bce304..1ae5c950f0 100755 --- a/configure +++ b/configure @@ -2082,6 +2082,13 @@ probe_target_compiler() { fi } +probe_target_compilers() { + for i; do + probe_target_compiler $i + test -n "$target_cc" && return 0 + done +} + write_target_makefile() { if test -n "$target_cc"; then echo "CC=$target_cc" @@ -2190,6 +2197,10 @@ if test "$QEMU_GA_VERSION" = ""; then QEMU_GA_VERSION=$(cat $source_path/VERSION) fi + +####################################### +# cross-compiled firmware targets + # Set up build tree symlinks that point back into the source tree # (these can be both files and directories). # Caution: avoid adding files or directories here using wildcards. This @@ -2216,19 +2227,27 @@ done # Mac OS X ships with a broken assembler roms= -if { test "$cpu" = "i386" || test "$cpu" = "x86_64"; } && \ +probe_target_compilers i386 x86_64 +if test -n "$target_cc" && test "$targetos" != "darwin" && test "$targetos" != "sunos" && \ test "$targetos" != "haiku" && test "$softmmu" = yes ; then # Different host OS linkers have different ideas about the name of the ELF # emulation. Linux and OpenBSD/amd64 use 'elf_i386'; FreeBSD uses the _fbsd # variant; OpenBSD/i386 uses the _obsd variant; and Windows uses i386pe. for emu in elf_i386 elf_i386_fbsd elf_i386_obsd i386pe; do - if "$ld" -verbose 2>&1 | grep -q "^[[:space:]]*$emu[[:space:]]*$"; then + if "$target_ld" -verbose 2>&1 | grep -q "^[[:space:]]*$emu[[:space:]]*$"; then ld_i386_emulation="$emu" - roms="optionrom" break fi done + if test -n "$ld_i386_emulation"; then + roms="optionrom" + config_mak=pc-bios/optionrom/config.mak + echo "# Automatically generated by configure - do not modify" > $config_mak + echo "TOPSRC_DIR=$source_path" >> $config_mak + echo "LD_I386_EMULATION=$ld_i386_emulation" >> $config_mak + write_target_makefile >> $config_mak + fi fi # Only build s390-ccw bios if the compiler has -march=z900 or -march=z10 @@ -2381,7 +2400,6 @@ echo "GLIB_LIBS=$glib_libs" >> $config_host_mak echo "GLIB_BINDIR=$glib_bindir" >> $config_host_mak echo "GLIB_VERSION=$(pkg-config --modversion glib-2.0)" >> $config_host_mak echo "QEMU_LDFLAGS=$QEMU_LDFLAGS" >> $config_host_mak -echo "LD_I386_EMULATION=$ld_i386_emulation" >> $config_host_mak echo "STRIP=$strip" >> $config_host_mak echo "EXESUF=$EXESUF" >> $config_host_mak @@ -2571,10 +2589,6 @@ for target in $target_list; do done echo "TCG_TESTS_TARGETS=$tcg_tests_targets" >> $makefile) -config_mak=pc-bios/optionrom/config.mak -echo "# Automatically generated by configure - do not modify" > $config_mak -echo "TOPSRC_DIR=$source_path" >> $config_mak - if test "$skip_meson" = no; then cross="config-meson.cross.new" meson_quote() { diff --git a/pc-bios/optionrom/Makefile b/pc-bios/optionrom/Makefile index 17ccc76241..f639915b4f 100644 --- a/pc-bios/optionrom/Makefile +++ b/pc-bios/optionrom/Makefile @@ -6,7 +6,6 @@ all: multiboot.bin multiboot_dma.bin linuxboot.bin linuxboot_dma.bin kvmvapic.bi # Dummy command so that make thinks it has done something @true -include ../../config-host.mak CFLAGS = -O2 -g quiet-command = $(if $(V),$1,$(if $(2),@printf " %-7s %s\n" $2 $3 && $1, @$1)) @@ -44,7 +43,6 @@ Wa = -Wa, override ASFLAGS += -32 override CFLAGS += $(call cc-option, $(Wa)-32) -LD_I386_EMULATION ?= elf_i386 override LDFLAGS = -m $(LD_I386_EMULATION) -T $(SRC_DIR)/flat.lds pvh.img: pvh.o pvh_main.o From d695918f7bd92604e561b6963e4ca9aef83f489b Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Fri, 27 May 2022 16:35:56 +0100 Subject: [PATCH 26/33] configure: enable cross compilation of vof MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit While container-based cross compilers are not supported, this already makes it possible to build vof on any machine that has an installation of GCC and binutils for 32- or 64-bit PowerPC. Reviewed-by: Richard Henderson Signed-off-by: Paolo Bonzini Message-Id: <20220517092616.1272238-16-pbonzini@redhat.com> Signed-off-by: Alex Bennée Message-Id: <20220527153603.887929-27-alex.bennee@linaro.org> --- configure | 10 ++++++++++ pc-bios/vof/Makefile | 17 +++++++++-------- 2 files changed, 19 insertions(+), 8 deletions(-) diff --git a/configure b/configure index 1ae5c950f0..11e7f30553 100755 --- a/configure +++ b/configure @@ -2213,6 +2213,7 @@ LINKS="Makefile" LINKS="$LINKS tests/tcg/Makefile.target" LINKS="$LINKS pc-bios/optionrom/Makefile" LINKS="$LINKS pc-bios/s390-ccw/Makefile" +LINKS="$LINKS pc-bios/vof/Makefile" LINKS="$LINKS .gdbinit scripts" # scripts needed by relative path in .gdbinit LINKS="$LINKS tests/avocado tests/data" LINKS="$LINKS tests/qemu-iotests/check" @@ -2250,6 +2251,15 @@ if test -n "$target_cc" && fi fi +probe_target_compilers ppc ppc64 +if test -n "$target_cc" && test "$softmmu" = yes; then + roms="$roms vof" + config_mak=pc-bios/vof/config.mak + echo "# Automatically generated by configure - do not modify" > $config_mak + echo "SRC_DIR=$source_path/pc-bios/vof" >> $config_mak + write_target_makefile >> $config_mak +fi + # Only build s390-ccw bios if the compiler has -march=z900 or -march=z10 # (which is the lowest architecture level that Clang supports) probe_target_compiler s390x diff --git a/pc-bios/vof/Makefile b/pc-bios/vof/Makefile index aa1678c4d8..391ac0d600 100644 --- a/pc-bios/vof/Makefile +++ b/pc-bios/vof/Makefile @@ -1,11 +1,10 @@ -all: build-all +include config.mak +VPATH=$(SRC_DIR) +all: vof.bin -build-all: vof.bin - -CROSS ?= -CC = $(CROSS)gcc -LD = $(CROSS)ld -OBJCOPY = $(CROSS)objcopy +CC ?= $(CROSS)gcc +LD ?= $(CROSS)ld +OBJCOPY ?= $(CROSS)objcopy %.o: %.S $(CC) -m32 -mbig-endian -mcpu=power4 -c -o $@ $< @@ -14,10 +13,12 @@ OBJCOPY = $(CROSS)objcopy $(CC) -m32 -mbig-endian -mcpu=power4 -c -fno-stack-protector -o $@ $< vof.elf: entry.o main.o ci.o bootmem.o libc.o - $(LD) -nostdlib -e_start -Tvof.lds -EB -o $@ $^ + $(LD) -nostdlib -e_start -T$(SRC_DIR)/vof.lds -EB -o $@ $^ %.bin: %.elf $(OBJCOPY) -O binary -j .text -j .data -j .toc -j .got2 $^ $@ clean: rm -f *.o vof.bin vof.elf *~ + +.PHONY: all clean From f21db42631ae1f3feaa3bf94edd9c06c24b4aeb6 Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Fri, 27 May 2022 16:35:57 +0100 Subject: [PATCH 27/33] configure: remove unused variables from config-host.mak MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The only compiler variable that is still needed is $(CC), for contrib/plugins/Makefile. All firmware builds have their own config-host.mak file. Signed-off-by: Paolo Bonzini Message-Id: <20220517092616.1272238-17-pbonzini@redhat.com> Signed-off-by: Alex Bennée Message-Id: <20220527153603.887929-28-alex.bennee@linaro.org> --- configure | 6 ------ 1 file changed, 6 deletions(-) diff --git a/configure b/configure index 11e7f30553..b9ccff9067 100755 --- a/configure +++ b/configure @@ -2397,11 +2397,6 @@ echo "GENISOIMAGE=$genisoimage" >> $config_host_mak echo "MESON=$meson" >> $config_host_mak echo "NINJA=$ninja" >> $config_host_mak echo "CC=$cc" >> $config_host_mak -echo "AR=$ar" >> $config_host_mak -echo "AS=$as" >> $config_host_mak -echo "CCAS=$ccas" >> $config_host_mak -echo "OBJCOPY=$objcopy" >> $config_host_mak -echo "LD=$ld" >> $config_host_mak echo "QEMU_CFLAGS=$QEMU_CFLAGS" >> $config_host_mak echo "QEMU_CXXFLAGS=$QEMU_CXXFLAGS" >> $config_host_mak echo "QEMU_OBJCFLAGS=$QEMU_OBJCFLAGS" >> $config_host_mak @@ -2410,7 +2405,6 @@ echo "GLIB_LIBS=$glib_libs" >> $config_host_mak echo "GLIB_BINDIR=$glib_bindir" >> $config_host_mak echo "GLIB_VERSION=$(pkg-config --modversion glib-2.0)" >> $config_host_mak echo "QEMU_LDFLAGS=$QEMU_LDFLAGS" >> $config_host_mak -echo "STRIP=$strip" >> $config_host_mak echo "EXESUF=$EXESUF" >> $config_host_mak # use included Linux headers From 6a0e7ea7b87cfa684bbcebaebafaef14c29a8872 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 27 May 2022 16:35:58 +0100 Subject: [PATCH 28/33] gitlab: introduce a common base job template MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Currently job rules are spread across the various templates and jobs, making it hard to understand exactly what runs in what scenario. This leads to inconsistency in the rules and increased maint burden. The intent is that we introduce a common '.base_job_template' which will have a general purpose 'rules:' block. No other template or job should define 'rules:', but instead they must rely on the inherited rules. To allow behaviour to be tweaked, rules will be influenced by a number of variables with the naming scheme 'QEMU_JOB_nnnn'. Signed-off-by: Daniel P. Berrangé Message-Id: <20220526110705.59952-2-berrange@redhat.com> Signed-off-by: Alex Bennée Reviewed-by: Thomas Huth Message-Id: <20220527153603.887929-29-alex.bennee@linaro.org> --- .gitlab-ci.d/base.yml | 28 +++++++++++++++++++++++++++ .gitlab-ci.d/qemu-project.yml | 1 + docs/devel/ci-jobs.rst.inc | 36 ++++++++++++++++++++++++++++++++++- 3 files changed, 64 insertions(+), 1 deletion(-) create mode 100644 .gitlab-ci.d/base.yml diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml new file mode 100644 index 0000000000..10eb6ab8bc --- /dev/null +++ b/.gitlab-ci.d/base.yml @@ -0,0 +1,28 @@ + +# The order of rules defined here is critically important. +# They are evaluated in order and first match wins. +# +# Thus we group them into a number of stages, ordered from +# most restrictive to least restrictive +# +.base_job_template: + rules: + ############################################################# + # Stage 1: exclude scenarios where we definitely don't + # want jobs to run + ############################################################# + + + ############################################################# + # Stage 2: fine tune execution of jobs in specific scenarios + # where the catch all logic is inapprorpaite + ############################################################# + + + ############################################################# + # Stage 3: catch all logic applying to any job not matching + # an earlier criteria + ############################################################# + + # Jobs can run if any jobs they depend on were successfull + - when: on_success diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml index 871262fe0e..691d9bf5dc 100644 --- a/.gitlab-ci.d/qemu-project.yml +++ b/.gitlab-ci.d/qemu-project.yml @@ -2,6 +2,7 @@ # https://gitlab.com/qemu-project/qemu/-/pipelines include: + - local: '/.gitlab-ci.d/base.yml' - local: '/.gitlab-ci.d/stages.yml' - local: '/.gitlab-ci.d/edk2.yml' - local: '/.gitlab-ci.d/opensbi.yml' diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc index 92e25872aa..eb6a9e6122 100644 --- a/docs/devel/ci-jobs.rst.inc +++ b/docs/devel/ci-jobs.rst.inc @@ -28,7 +28,35 @@ For further information about how to set these variables, please refer to:: https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd -Here is a list of the most used variables: +Variable naming and grouping +---------------------------- + +The variables used by QEMU's CI configuration are grouped together +in a handful of namespaces + + * QEMU_JOB_nnnn - variables to be defined in individual jobs + or templates, to influence the shared rules defined in the + .base_job_template. + + * QEMU_CI_nnn - variables to be set by contributors in their + repository CI settings, or as git push variables, to influence + which jobs get run in a pipeline + + * nnn - other misc variables not falling into the above + categories, or using different names for historical reasons + and not yet converted. + +Maintainer controlled job variables +----------------------------------- + +The following variables may be set when defining a job in the +CI configuration file. + +Contributor controlled runtime variables +---------------------------------------- + +The following variables may be set by contributors to control +job execution QEMU_CI_AVOCADO_TESTING ~~~~~~~~~~~~~~~~~~~~~~~ @@ -38,6 +66,12 @@ these artifacts are not already cached, downloading them make the jobs reach the timeout limit). Set this variable to have the tests using the Avocado framework run automatically. +Other misc variables +-------------------- + +These variables are primarily to control execution of jobs on +private runners + AARCH64_RUNNER_AVAILABLE ~~~~~~~~~~~~~~~~~~~~~~~~ If you've got access to an aarch64 host that can be used as a gitlab-CI From 00125414ba1b9d04dfec3af6fb75e39591b53892 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 27 May 2022 16:35:59 +0100 Subject: [PATCH 29/33] gitlab: convert Cirrus jobs to .base_job_template MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This folds the Cirrus job rules into the base job template, introducing two new variables - QEMU_JOB_CIRRUS - identifies the job as making use of Cirrus CI via cirrus-run - QEMU_JOB_OPTIONAL - identifies the job as one that is not run by default, primarily due to resource constraints. It can be manually invoked by users if they wish to validate that scenario. Signed-off-by: Daniel P. Berrangé Message-Id: <20220526110705.59952-3-berrange@redhat.com> Signed-off-by: Alex Bennée Reviewed-by: Thomas Huth Message-Id: <20220527153603.887929-30-alex.bennee@linaro.org> --- .gitlab-ci.d/base.yml | 9 +++++++++ .gitlab-ci.d/cirrus.yml | 16 ++++++++-------- docs/devel/ci-jobs.rst.inc | 14 ++++++++++++++ 3 files changed, 31 insertions(+), 8 deletions(-) diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml index 10eb6ab8bc..5734caf9fe 100644 --- a/.gitlab-ci.d/base.yml +++ b/.gitlab-ci.d/base.yml @@ -12,12 +12,21 @@ # want jobs to run ############################################################# + # Cirrus jobs can't run unless the creds / target repo are set + - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == "" || $CIRRUS_API_TOKEN == "")' + when: never + ############################################################# # Stage 2: fine tune execution of jobs in specific scenarios # where the catch all logic is inapprorpaite ############################################################# + # Optional jobs should not be run unless manually triggered + - if: '$QEMU_JOB_OPTIONAL' + when: manual + allow_failure: true + ############################################################# # Stage 3: catch all logic applying to any job not matching diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml index b96b22e269..609c364308 100644 --- a/.gitlab-ci.d/cirrus.yml +++ b/.gitlab-ci.d/cirrus.yml @@ -11,6 +11,7 @@ # special care, because we can't just override it at the GitLab CI job # definition level or we risk breaking it completely. .cirrus_build_job: + extends: .base_job_template stage: build image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master needs: [] @@ -40,11 +41,8 @@ <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml - cat .gitlab-ci.d/cirrus/$NAME.yml - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml - rules: - # Allow on 'staging' branch and 'stable-X.Y-staging' branches only - - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH !~ /staging/' - when: never - - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN" + variables: + QEMU_JOB_CIRRUS: 1 x64-freebsd-12-build: extends: .cirrus_build_job @@ -90,11 +88,11 @@ x64-macos-11-base-build: # The following jobs run VM-based tests via KVM on a Linux-based Cirrus-CI job .cirrus_kvm_job: + extends: .base_job_template stage: build image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master needs: [] timeout: 80m - allow_failure: true script: - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g" -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g" @@ -105,8 +103,10 @@ x64-macos-11-base-build: <.gitlab-ci.d/cirrus/kvm-build.yml >.gitlab-ci.d/cirrus/$NAME.yml - cat .gitlab-ci.d/cirrus/$NAME.yml - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml - rules: - - when: manual + variables: + QEMU_JOB_CIRRUS: 1 + QEMU_JOB_OPTIONAL: 1 + x86-netbsd: extends: .cirrus_kvm_job diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc index eb6a9e6122..a539f502da 100644 --- a/docs/devel/ci-jobs.rst.inc +++ b/docs/devel/ci-jobs.rst.inc @@ -52,6 +52,20 @@ Maintainer controlled job variables The following variables may be set when defining a job in the CI configuration file. +QEMU_JOB_CIRRUS +~~~~~~~~~~~~~~~ + +The job makes use of Cirrus CI infrastructure, requiring the +configuration setup for cirrus-run to be present in the repository + +QEMU_JOB_OPTIONAL +~~~~~~~~~~~~~~~~~ + +The job is expected to be successful in general, but is not run +by default due to need to conserve limited CI resources. It is +available to be started manually by the contributor in the CI +pipelines UI. + Contributor controlled runtime variables ---------------------------------------- From 16fee101d9011ea5dfd56a055d19df802e59eb29 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 27 May 2022 16:36:00 +0100 Subject: [PATCH 30/33] gitlab: convert static checks to .base_job_template MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This folds the static checks into using the base job template rules, introducing one new variable - QEMU_JOB_ONLY_FORKS - a job that should never run on an upstream pipeline. The information it reports is only applicable to contributors in a pre-submission scenario, not time of merge. Signed-off-by: Daniel P. Berrangé Message-Id: <20220526110705.59952-4-berrange@redhat.com> [AJB: fix typo] Signed-off-by: Alex Bennée Reviewed-by: Thomas Huth Message-Id: <20220527153603.887929-31-alex.bennee@linaro.org> --- .gitlab-ci.d/base.yml | 4 ++++ .gitlab-ci.d/static_checks.yml | 19 +++++++------------ docs/devel/ci-jobs.rst.inc | 7 +++++++ 3 files changed, 18 insertions(+), 12 deletions(-) diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml index 5734caf9fe..e6953c77ae 100644 --- a/.gitlab-ci.d/base.yml +++ b/.gitlab-ci.d/base.yml @@ -16,6 +16,10 @@ - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == "" || $CIRRUS_API_TOKEN == "")' when: never + # Jobs only intended for forks should always be skipped on upstream + - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"' + when: never + ############################################################# # Stage 2: fine tune execution of jobs in specific scenarios diff --git a/.gitlab-ci.d/static_checks.yml b/.gitlab-ci.d/static_checks.yml index 94858e3272..289ad1359e 100644 --- a/.gitlab-ci.d/static_checks.yml +++ b/.gitlab-ci.d/static_checks.yml @@ -1,4 +1,5 @@ check-patch: + extends: .base_job_template stage: build image: python:3.10-alpine needs: [] @@ -6,15 +7,13 @@ check-patch: - .gitlab-ci.d/check-patch.py variables: GIT_DEPTH: 1000 + QEMU_JOB_ONLY_FORKS: 1 before_script: - apk -U add git perl - rules: - - if: '$CI_PROJECT_NAMESPACE == "qemu-project"' - when: never - - when: on_success - allow_failure: true + allow_failure: true check-dco: + extends: .base_job_template stage: build image: python:3.10-alpine needs: [] @@ -23,12 +22,9 @@ check-dco: GIT_DEPTH: 1000 before_script: - apk -U add git - rules: - - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH' - when: never - - when: on_success check-python-pipenv: + extends: .base_job_template stage: test image: $CI_REGISTRY_IMAGE/qemu/python:latest script: @@ -39,6 +35,7 @@ check-python-pipenv: job: python-container check-python-tox: + extends: .base_job_template stage: test image: $CI_REGISTRY_IMAGE/qemu/python:latest script: @@ -46,8 +43,6 @@ check-python-tox: variables: GIT_DEPTH: 1 QEMU_TOX_EXTRA_ARGS: --skip-missing-interpreters=false + QEMU_JOB_OPTIONAL: 1 needs: job: python-container - rules: - - when: manual - allow_failure: true diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc index a539f502da..4c7e30ab08 100644 --- a/docs/devel/ci-jobs.rst.inc +++ b/docs/devel/ci-jobs.rst.inc @@ -66,6 +66,13 @@ by default due to need to conserve limited CI resources. It is available to be started manually by the contributor in the CI pipelines UI. +QEMU_JOB_ONLY_FORKS +~~~~~~~~~~~~~~~~~~~ + +The job results are only of interest to contributors prior to +submitting code. They are not required as part of the gating +CI pipeline. + Contributor controlled runtime variables ---------------------------------------- From e312d1fdbbb3a53ee30ab84203344588154129f9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 27 May 2022 16:36:01 +0100 Subject: [PATCH 31/33] gitlab: convert build/container jobs to .base_job_template MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This converts the main build and container jobs to use the base job rules, defining the following new variables - QEMU_JOB_SKIPPED - jobs that are known to be currently broken and should not be run. Can still be manually launched if desired. - QEMU_JOB_AVOCADO - jobs that run the Avocado integration test harness. - QEMU_JOB_PUBLISH - jobs that publish content after the branch is merged upstream As build-tools-and-docs runs on master we declare the requirement of building amd64-debian-container optional as it should already exits once we merge. Signed-off-by: Daniel P. Berrangé Message-Id: <20220526110705.59952-5-berrange@redhat.com> [AJB: fix upstream typo, mention optional container req] Signed-off-by: Alex Bennée Reviewed-by: Thomas Huth Message-Id: <20220527153603.887929-32-alex.bennee@linaro.org> --- .gitlab-ci.d/base.yml | 22 ++++++++++++++++++++++ .gitlab-ci.d/buildtest-template.yml | 16 ++++------------ .gitlab-ci.d/buildtest.yml | 28 +++++++++++++--------------- .gitlab-ci.d/container-cross.yml | 6 ++---- .gitlab-ci.d/container-template.yml | 1 + .gitlab-ci.d/crossbuild-template.yml | 3 +++ .gitlab-ci.d/windows.yml | 1 + docs/devel/ci-jobs.rst.inc | 19 +++++++++++++++++++ 8 files changed, 65 insertions(+), 31 deletions(-) diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml index e6953c77ae..4f091d5aad 100644 --- a/.gitlab-ci.d/base.yml +++ b/.gitlab-ci.d/base.yml @@ -16,10 +16,22 @@ - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == "" || $CIRRUS_API_TOKEN == "")' when: never + # Publishing jobs should only run on the default branch in upstream + - if: '$QEMU_JOB_PUBLISH == "1" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH' + when: never + + # Non-publishing jobs should only run on staging branches in upstream + - if: '$QEMU_JOB_PUBLISH != "1" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH !~ /staging/' + when: never + # Jobs only intended for forks should always be skipped on upstream - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"' when: never + # Avocado jobs don't run in forks unless $QEMU_CI_AVOCADO_TESTING is set + - if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" && $CI_PROJECT_NAMESPACE != "qemu-project"' + when: never + ############################################################# # Stage 2: fine tune execution of jobs in specific scenarios @@ -31,6 +43,16 @@ when: manual allow_failure: true + # Skipped jobs should not be run unless manually triggered + - if: '$QEMU_JOB_SKIPPED' + when: manual + allow_failure: true + + # Avocado jobs can be manually start in forks if $QEMU_CI_AVOCADO_TESTING is unset + - if: '$QEMU_JOB_AVOCADO && $CI_PROJECT_NAMESPACE != "qemu-project"' + when: manual + allow_failure: true + ############################################################# # Stage 3: catch all logic applying to any job not matching diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml index b381345dbc..73ecfabb8d 100644 --- a/.gitlab-ci.d/buildtest-template.yml +++ b/.gitlab-ci.d/buildtest-template.yml @@ -1,4 +1,5 @@ .native_build_job_template: + extends: .base_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest before_script: @@ -27,6 +28,7 @@ fi .common_test_job_template: + extends: .base_job_template stage: test image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script: @@ -77,15 +79,5 @@ after_script: - cd build - du -chs ${CI_PROJECT_DIR}/avocado-cache - rules: - # Only run these jobs if running on the mainstream namespace, - # or if the user set the QEMU_CI_AVOCADO_TESTING variable (either - # in its namespace setting or via git-push option, see documentation - # in /.gitlab-ci.yml of this repository). - - if: '$CI_PROJECT_NAMESPACE == "qemu-project"' - when: on_success - - if: '$QEMU_CI_AVOCADO_TESTING' - when: on_success - # Otherwise, set to manual (the jobs are created but not run). - - when: manual - allow_failure: true + variables: + QEMU_JOB_AVOCADO: 1 diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml index e9620c3074..ecac3ec50c 100644 --- a/.gitlab-ci.d/buildtest.yml +++ b/.gitlab-ci.d/buildtest.yml @@ -360,12 +360,11 @@ build-cfi-aarch64: expire_in: 2 days paths: - build - rules: + variables: # FIXME: This job is often failing, likely due to out-of-memory problems in # the constrained containers of the shared runners. Thus this is marked as - # manual until the situation has been solved. - - when: manual - allow_failure: true + # skipped until the situation has been solved. + QEMU_JOB_SKIPPED: 1 check-cfi-aarch64: extends: .native_test_job_template @@ -402,12 +401,11 @@ build-cfi-ppc64-s390x: expire_in: 2 days paths: - build - rules: + variables: # FIXME: This job is often failing, likely due to out-of-memory problems in # the constrained containers of the shared runners. Thus this is marked as - # manual until the situation has been solved. - - when: manual - allow_failure: true + # skipped until the situation has been solved. + QEMU_JOB_SKIPPED: 1 check-cfi-ppc64-s390x: extends: .native_test_job_template @@ -579,6 +577,7 @@ build-without-default-features: MAKE_CHECK_ARGS: check-unit check-qtest SPEED=slow build-libvhost-user: + extends: .base_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/fedora:latest needs: @@ -595,10 +594,13 @@ build-tools-and-docs-debian: extends: .native_build_job_template needs: job: amd64-debian-container + # when running on 'master' we use pre-existing container + optional: true variables: IMAGE: debian-amd64 MAKE_CHECK_ARGS: check-unit check-softfloat ctags TAGS cscope CONFIGURE_ARGS: --disable-system --disable-user --enable-docs --enable-tools + QEMU_JOB_PUBLISH: 1 artifacts: expire_in: 2 days paths: @@ -618,6 +620,7 @@ build-tools-and-docs-debian: # that users can see the results of their commits, regardless # of what topic branch they're currently using pages: + extends: .base_job_template image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest stage: test needs: @@ -635,10 +638,5 @@ pages: artifacts: paths: - public - rules: - - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH' - when: on_success - - if: '$CI_PROJECT_NAMESPACE == "qemu-project"' - when: never - - if: '$CI_PROJECT_NAMESPACE != "qemu-project"' - when: on_success + variables: + QEMU_JOB_PUBLISH: 1 diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml index 147e667744..b7963498a3 100644 --- a/.gitlab-ci.d/container-cross.yml +++ b/.gitlab-ci.d/container-cross.yml @@ -40,15 +40,13 @@ armhf-debian-cross-container: # We never want to build hexagon in the CI system and by default we # always want to refer to the master registry where it lives. hexagon-cross-container: + extends: .base_job_template image: docker:stable stage: containers - rules: - - if: '$CI_PROJECT_NAMESPACE == "qemu-project"' - when: never - - when: always variables: NAME: debian-hexagon-cross GIT_DEPTH: 1 + QEMU_JOB_ONLY_FORKS: 1 services: - docker:dind before_script: diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml index 1baecd9460..c434b9c8f3 100644 --- a/.gitlab-ci.d/container-template.yml +++ b/.gitlab-ci.d/container-template.yml @@ -1,4 +1,5 @@ .container_job_template: + extends: .base_job_template image: docker:stable stage: containers services: diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml index 29c3c2b826..28b2142ec2 100644 --- a/.gitlab-ci.d/crossbuild-template.yml +++ b/.gitlab-ci.d/crossbuild-template.yml @@ -1,4 +1,5 @@ .cross_system_build_job: + extends: .base_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest timeout: 80m @@ -24,6 +25,7 @@ # KVM), and set extra options (such disabling other accelerators) via the # $EXTRA_CONFIGURE_OPTS variable. .cross_accel_build_job: + extends: .base_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest timeout: 30m @@ -36,6 +38,7 @@ - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS .cross_user_build_job: + extends: .base_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script: diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml index cf7724b8e5..1b2ede49e1 100644 --- a/.gitlab-ci.d/windows.yml +++ b/.gitlab-ci.d/windows.yml @@ -1,4 +1,5 @@ .shared_msys2_builder: + extends: .base_job_template tags: - shared-windows - windows diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc index 4c7e30ab08..0b4926e537 100644 --- a/docs/devel/ci-jobs.rst.inc +++ b/docs/devel/ci-jobs.rst.inc @@ -73,6 +73,25 @@ The job results are only of interest to contributors prior to submitting code. They are not required as part of the gating CI pipeline. +QEMU_JOB_SKIPPED +~~~~~~~~~~~~~~~~ + +The job is not reliably successsful in general, so is not +currently suitable to be run by default. Ideally this should +be a temporary marker until the problems can be addressed, or +the job permanently removed. + +QEMU_JOB_PUBLISH +~~~~~~~~~~~~~~~~ + +The job is for publishing content after a branch has been +merged into the upstream default branch. + +QEMU_JOB_AVOCADO +~~~~~~~~~~~~~~~~ + +The job runs the Avocado integration test suite + Contributor controlled runtime variables ---------------------------------------- From 28357dc525b4798cdef1101cbb459afcd7233280 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 27 May 2022 16:36:02 +0100 Subject: [PATCH 32/33] gitlab: don't run CI jobs in forks by default MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit To preserve CI shared runner credits we don't want to run pipelines on every push. This sets up the config so that pipelines are never created for contributors by default. To override this the QEMU_CI variable can be set to a non-zero value. If set to 1, the pipeline will be created but all jobs will remain manually started. The contributor can selectively run jobs that they care about. If set to 2, the pipeline will be created and all jobs will immediately start. This behavior can be controlled using push variables git push -o ci.variable=QEMU_CI=1 To make this more convenient define an alias git config --local alias.push-ci "push -o ci.variable=QEMU_CI=1" git config --local alias.push-ci-now "push -o ci.variable=QEMU_CI=2" Which lets you run git push-ci to create the pipeline, or git push-ci-now to create and run the pipeline Signed-off-by: Daniel P. Berrangé Message-Id: <20220526110705.59952-6-berrange@redhat.com> [AJB: fix typo, replicate alias tips in ci.rst] Signed-off-by: Alex Bennée Reviewed-by: Thomas Huth Message-Id: <20220527153603.887929-33-alex.bennee@linaro.org> --- .gitlab-ci.d/base.yml | 9 +++++++++ docs/devel/ci-jobs.rst.inc | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+) diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml index 4f091d5aad..f334f3ded7 100644 --- a/.gitlab-ci.d/base.yml +++ b/.gitlab-ci.d/base.yml @@ -28,6 +28,10 @@ - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"' when: never + # Forks don't get pipelines unless QEMU_CI=1 or QEMU_CI=2 is set + - if: '$QEMU_CI != "1" && $QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"' + when: never + # Avocado jobs don't run in forks unless $QEMU_CI_AVOCADO_TESTING is set - if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" && $CI_PROJECT_NAMESPACE != "qemu-project"' when: never @@ -59,5 +63,10 @@ # an earlier criteria ############################################################# + # Forks pipeline jobs don't start automatically unless + # QEMU_CI=2 is set + - if: '$QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"' + when: manual + # Jobs can run if any jobs they depend on were successfull - when: on_success diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc index 0b4926e537..9118a61a17 100644 --- a/docs/devel/ci-jobs.rst.inc +++ b/docs/devel/ci-jobs.rst.inc @@ -28,6 +28,32 @@ For further information about how to set these variables, please refer to:: https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd +Setting aliases in your git config +---------------------------------- + +You can use aliases to make it easier to push branches with different +CI configurations. For example define an alias for triggering CI: + +.. code:: + + git config --local alias.push-ci "push -o ci.variable=QEMU_CI=1" + git config --local alias.push-ci-now "push -o ci.variable=QEMU_CI=2" + +Which lets you run: + +.. code:: + + git push-ci + +to create the pipeline, or: + +.. code:: + + git push-ci-now + +to create and run the pipeline + + Variable naming and grouping ---------------------------- @@ -98,6 +124,18 @@ Contributor controlled runtime variables The following variables may be set by contributors to control job execution +QEMU_CI +~~~~~~~ + +By default, no pipelines will be created on contributor forks +in order to preserve CI credits + +Set this variable to 1 to create the pipelines, but leave all +the jobs to be manually started from the UI + +Set this variable to 2 to create the pipelines and run all +the jobs immediately, as was historicaly behaviour + QEMU_CI_AVOCADO_TESTING ~~~~~~~~~~~~~~~~~~~~~~~ By default, tests using the Avocado framework are not run automatically in From 7266ecce502c31387a3cbf83d7297bc9cf27b139 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alex=20Benn=C3=A9e?= Date: Fri, 27 May 2022 16:36:03 +0100 Subject: [PATCH 33/33] docs/devel: clean-up the CI links in the docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit There where some broken links so fix those up with proper references to the devel docs. I also did a little light copy-editing to reflect the current state and broke up a paragraph to reduce the "wall of text" effect. Signed-off-by: Alex Bennée Reviewed-by: Daniel P. Berrangé Message-Id: <20220527153603.887929-34-alex.bennee@linaro.org> --- docs/devel/ci-jobs.rst.inc | 2 ++ docs/devel/ci.rst | 11 +++++----- docs/devel/submitting-a-patch.rst | 36 ++++++++++++++++--------------- docs/devel/testing.rst | 2 ++ 4 files changed, 29 insertions(+), 22 deletions(-) diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc index 9118a61a17..1f28fec0d0 100644 --- a/docs/devel/ci-jobs.rst.inc +++ b/docs/devel/ci-jobs.rst.inc @@ -1,3 +1,5 @@ +.. _ci_var: + Custom CI/CD variables ====================== diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst index d106610096..ed88a2010b 100644 --- a/docs/devel/ci.rst +++ b/docs/devel/ci.rst @@ -1,12 +1,13 @@ +.. _ci: + == CI == -QEMU has configurations enabled for a number of different CI services. -The most up to date information about them and their status can be -found at:: - - https://wiki.qemu.org/Testing/CI +Most of QEMU's CI is run on GitLab's infrastructure although a number +of other CI services are used for specialised purposes. The most up to +date information about them and their status can be found on the +`project wiki testing page `_. .. include:: ci-definitions.rst.inc .. include:: ci-jobs.rst.inc diff --git a/docs/devel/submitting-a-patch.rst b/docs/devel/submitting-a-patch.rst index e51259eb9c..d3876ec1b7 100644 --- a/docs/devel/submitting-a-patch.rst +++ b/docs/devel/submitting-a-patch.rst @@ -204,23 +204,25 @@ log`` for these keywords for example usage. Test your patches ~~~~~~~~~~~~~~~~~ -Although QEMU has `continuous integration -services `__ that attempt to test -patches submitted to the list, it still saves everyone time if you have -already tested that your patch compiles and works. Because QEMU is such -a large project, it's okay to use configure arguments to limit what is -built for faster turnaround during your development time; but it is -still wise to also check that your patches work with a full build before -submitting a series, especially if your changes might have an unintended -effect on other areas of the code you don't normally experiment with. -See `Testing `__ for more details on what tests are available. -Also, it is a wise idea to include a testsuite addition as part of your -patches - either to ensure that future changes won't regress your new -feature, or to add a test which exposes the bug that the rest of your -series fixes. Keeping separate commits for the test and the fix allows -reviewers to rebase the test to occur first to prove it catches the -problem, then again to place it last in the series so that bisection -doesn't land on a known-broken state. +Although QEMU uses various :ref:`ci` services that attempt to test +patches submitted to the list, it still saves everyone time if you +have already tested that your patch compiles and works. Because QEMU +is such a large project the default configuration won't create a +testing pipeline on GitLab when a branch is pushed. See the :ref:`CI +variable documentation` for details on how to control the +running of tests; but it is still wise to also check that your patches +work with a full build before submitting a series, especially if your +changes might have an unintended effect on other areas of the code you +don't normally experiment with. See :ref:`testing` for more details on +what tests are available. + +Also, it is a wise idea to include a testsuite addition as part of +your patches - either to ensure that future changes won't regress your +new feature, or to add a test which exposes the bug that the rest of +your series fixes. Keeping separate commits for the test and the fix +allows reviewers to rebase the test to occur first to prove it catches +the problem, then again to place it last in the series so that +bisection doesn't land on a known-broken state. .. _submitting_your_patches: diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index 5b60a31807..3f6ebd5073 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -1,3 +1,5 @@ +.. _testing: + Testing in QEMU ===============