Ubuntu 24.04 (#32624)

* noble build

* workflow

* symlink

* map only for noble

* sym

* add qt to cppcheck

* cppcheck noreturn in non void funct

* get kaitai header

* kaitai header

* syntax

* try new pyopencl

* try this version

* install lsb-core manually

* support old 20.04 lsb-core

* try arm

* try latest pyopencl

* revert

* use pocl icd

* no lock

* no arm pyopencl

* new intel opencl runtime

* pin filelock

* undo

* glsl version

* new version

* build test

* remove test

* new metadrive

* remove glsl restrictions

* <optional>

* Update .github/workflows/tools_tests.yaml

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* cache

* new cache

---------

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

.devcontainer/Dockerfile

@@ -1,11 +1,11 @@
 FROM ghcr.io/commaai/openpilot-base:latest
 
-RUN apt update && apt install -y vim net-tools usbutils htop ripgrep tmux wget mesa-utils xvfb libxtst6 libxv1 libglu1-mesa libegl1-mesa gdb bash-completion
+RUN apt update && apt install -y vim net-tools usbutils htop ripgrep tmux wget mesa-utils xvfb libxtst6 libxv1 libglu1-mesa gdb bash-completion
 RUN pip install ipython jupyter jupyterlab
 
 RUN cd /tmp && \
     ARCH=$(arch | sed s/aarch64/arm64/ | sed s/x86_64/amd64/) && \
-    curl -L -o virtualgl.deb "https://downloads.sourceforge.net/project/virtualgl/3.1/virtualgl_3.1_$ARCH.deb" && \
+    curl -L -o virtualgl.deb "https://github.com/VirtualGL/virtualgl/releases/download/3.1.1/virtualgl_3.1.1_$ARCH.deb" && \
     dpkg -i virtualgl.deb
 
 RUN usermod -aG video batman

.github/workflows/tools_tests.yaml

@@ -39,12 +39,13 @@ jobs:
                         source selfdrive/test/setup_vsound.sh && \
                         CI=1 pytest tools/sim/tests/test_metadrive_bridge.py"
 
-  test_python311:
-    name: test python3.11 support
-    runs-on: ubuntu-latest
-    timeout-minutes: 5
+  test_compatibility:
+    name: test 20.04 + 3.11
+    runs-on: ubuntu-20.04
     steps:
     - uses: actions/checkout@v4
+      with:
+        submodules: true
     - name: Installing ubuntu dependencies
       run: INSTALL_EXTRA_PACKAGES=no tools/install_ubuntu_dependencies.sh
     - name: Installing python
@@ -57,6 +58,25 @@ jobs:
       run: pip install poetry==1.7.0
     - name: Installing python dependencies
       run: poetry install --no-cache --no-root
+    - name: git LFS
+      run: git lfs pull
+    - run: echo "CACHE_COMMIT_DATE=$(git log -1 --pretty='format:%cd' --date=format:'%Y-%m-%d-%H:%M')" >> $GITHUB_ENV
+    - name: Getting scons cache
+      uses: 'actions/cache@v4'
+      with:
+        path: /tmp/scons_cache
+        key: scons-${{ runner.arch }}-ubuntu2004-${{ env.CACHE_COMMIT_DATE }}-${{ github.sha }}
+        restore-keys: |
+          scons-${{ runner.arch }}-ubuntu2004-${{ env.CACHE_COMMIT_DATE }}
+          scons-${{ runner.arch }}-ubuntu2004
+    - name: Building openpilot
+      run: poetry run scons -u -j$(nproc)
+    - name: Saving scons cache
+      uses: actions/cache/save@v4
+      if: github.ref == 'refs/heads/master'
+      with:
+        path: /tmp/scons_cache
+        key: scons-${{ runner.arch }}-ubuntu2004-${{ env.CACHE_COMMIT_DATE }}-${{ github.sha }}
 
   devcontainer:
     name: devcontainer

.pre-commit-config.yaml

@@ -62,6 +62,8 @@ repos:
         - --quiet
         - --force
         - -j8
+        - --library=qt
+        - --include=third_party/kaitai/kaitaistream.h
 -   repo: https://github.com/cpplint/cpplint
     rev: 1.6.1
     hooks:

Dockerfile.openpilot_base

@@ -1,4 +1,4 @@
-FROM ubuntu:20.04
+FROM ubuntu:24.04
 
 ENV PYTHONUNBUFFERED 1
 
@@ -15,7 +15,7 @@ ENV LC_ALL en_US.UTF-8
 COPY tools/install_ubuntu_dependencies.sh /tmp/tools/
 RUN INSTALL_EXTRA_PACKAGES=no /tmp/tools/install_ubuntu_dependencies.sh && \
     rm -rf /var/lib/apt/lists/* /tmp/* && \
-    cd /usr/lib/gcc/arm-none-eabi/9.2.1 && \
+    cd /usr/lib/gcc/arm-none-eabi/* && \
     rm -rf arm/ thumb/nofp thumb/v6* thumb/v8* thumb/v7+fp thumb/v7-r+fp.sp
 
 # Add OpenCL
@@ -30,23 +30,28 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     gcc-arm-none-eabi \
     tmux \
     vim \
-    lsb-core \
     libx11-6 \
+    wget \
   && rm -rf /var/lib/apt/lists/*
 
-ARG INTEL_DRIVER=l_opencl_p_18.1.0.015.tgz
-ARG INTEL_DRIVER_URL=https://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/15532
-RUN mkdir -p /tmp/opencl-driver-intel
-
-RUN cd /tmp/opencl-driver-intel && \
-    echo INTEL_DRIVER is $INTEL_DRIVER && \
-    curl -O $INTEL_DRIVER_URL/$INTEL_DRIVER && \
-    tar -xzf $INTEL_DRIVER && \
-    for i in $(basename $INTEL_DRIVER .tgz)/rpm/*.rpm; do alien --to-deb $i; done && \
-    dpkg -i *.deb && \
-    rm -rf $INTEL_DRIVER $(basename $INTEL_DRIVER .tgz) *.deb && \
+RUN mkdir -p /tmp/opencl-driver-intel && \
+    cd /tmp/opencl-driver-intel && \
+    wget https://github.com/intel/llvm/releases/download/2024-WW14/oclcpuexp-2024.17.3.0.09_rel.tar.gz && \
+    wget https://github.com/oneapi-src/oneTBB/releases/download/v2021.12.0/oneapi-tbb-2021.12.0-lin.tgz && \
+    mkdir -p /opt/intel/oclcpuexp_2024.17.3.0.09_rel && \
+    cd /opt/intel/oclcpuexp_2024.17.3.0.09_rel && \
+    tar -zxvf /tmp/opencl-driver-intel/oclcpuexp-2024.17.3.0.09_rel.tar.gz && \
     mkdir -p /etc/OpenCL/vendors && \
-    echo /opt/intel/opencl_compilers_and_libraries_18.1.0.015/linux/compiler/lib/intel64_lin/libintelocl.so > /etc/OpenCL/vendors/intel.icd && \
+    echo /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64/libintelocl.so > /etc/OpenCL/vendors/intel_expcpu.icd && \
+    cd /opt/intel && \
+    tar -zxvf /tmp/opencl-driver-intel/oneapi-tbb-2021.12.0-lin.tgz && \
+    ln -s /opt/intel/oneapi-tbb-2021.12.0/lib/intel64/gcc4.8/libtbb.so /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64 && \
+    ln -s /opt/intel/oneapi-tbb-2021.12.0/lib/intel64/gcc4.8/libtbbmalloc.so /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64 && \
+    ln -s /opt/intel/oneapi-tbb-2021.12.0/lib/intel64/gcc4.8/libtbb.so.12 /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64 && \
+    ln -s /opt/intel/oneapi-tbb-2021.12.0/lib/intel64/gcc4.8/libtbbmalloc.so.2 /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64 && \
+    mkdir -p /etc/ld.so.conf.d && \
+    echo /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64 > /etc/ld.so.conf.d/libintelopenclexp.conf && \
+    ldconfig -f /etc/ld.so.conf.d/libintelopenclexp.conf && \
     cd / && \
     rm -rf /tmp/opencl-driver-intel
 
@@ -57,7 +62,7 @@ ENV QTWEBENGINE_DISABLE_SANDBOX 1
 RUN dbus-uuidgen > /etc/machine-id
 
 ARG USER=batman
-ARG USER_UID=1000
+ARG USER_UID=1001
 RUN useradd -m -s /bin/bash -u $USER_UID $USER
 RUN usermod -aG sudo $USER
 RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers

SConstruct

@@ -14,6 +14,8 @@ SCons.Warnings.warningAsException(True)
 
 TICI = os.path.isfile('/TICI')
 AGNOS = TICI
+UBUNTU_FOCAL = int(subprocess.check_output('[ -f /etc/os-release ] && . /etc/os-release && [ "$ID" = "ubuntu" ] && [ "$VERSION_ID" = "20.04" ] && echo 1 || echo 0', shell=True, encoding='utf-8').rstrip())
+Export('UBUNTU_FOCAL')
 
 Decider('MD5-timestamp')
 

poetry.lock

@@ -2270,8 +2270,8 @@ ros = ["zmq"]
 [package.source]
 type = "git"
 url = "https://github.com/commaai/metadrive.git"
-reference = "python3.12"
-resolved_reference = "0823c7491a1ab870bcd53d10a41e985a3ad1b968"
+reference = "anisotropic_off"
+resolved_reference = "1606c844345f98aa79f76177856e328edc6ec428"
 
 [[package]]
 name = "mouseinfo"
@@ -8114,4 +8114,4 @@ files = [
 [metadata]
 lock-version = "2.0"
 python-versions = ">=3.11, <3.13"
-content-hash = "83dbbf76ea5f3e1022233a164d6f342706ebf64ad8bbbf742a55d5f49474d863"
+content-hash = "32df734a6ad6ad6a665c94c41cd0e3643a48174904faaf6f21c5c84f350f3cec"

pyproject.toml

@@ -176,7 +176,7 @@ flaky = "*"
 inputs = "*"
 lru-dict = "*"
 matplotlib = "*"
-metadrive-simulator = { git = "https://github.com/commaai/metadrive.git", branch = "python3.12", markers = "platform_machine != 'aarch64'" } # no linux/aarch64 wheels for certain dependencies
+metadrive-simulator = { git = "https://github.com/commaai/metadrive.git", branch = "anisotropic_off", markers = "platform_machine != 'aarch64'" } # no linux/aarch64 wheels for certain dependencies
 mpld3 = "*"
 myst-parser = "*"
 natsort = "*"

selfdrive/navd/SConscript

@@ -1,4 +1,4 @@
-Import('qt_env', 'arch', 'common', 'messaging', 'visionipc', 'transformations')
+Import('qt_env', 'arch', 'common', 'messaging', 'visionipc', 'transformations', 'UBUNTU_FOCAL')
 
 map_env = qt_env.Clone()
 libs = ['qt_widgets', 'qt_util', 'QMapLibre', common, messaging, visionipc, transformations,
@@ -6,7 +6,7 @@ libs = ['qt_widgets', 'qt_util', 'QMapLibre', common, messaging, visionipc, tran
 if arch == 'larch64':
   libs.append(':libEGL_mesa.so.0')
 
-if arch in ['larch64', 'aarch64', 'x86_64']:
+if arch in ['larch64', 'aarch64', 'x86_64'] and not UBUNTU_FOCAL:
   if arch == 'x86_64':
     rpath = Dir(f"#third_party/maplibre-native-qt/{arch}/lib").srcnode().abspath
     map_env["RPATH"] += [rpath, ]

selfdrive/ui/SConscript

@@ -1,6 +1,6 @@
 import os
 import json
-Import('qt_env', 'arch', 'common', 'messaging', 'visionipc', 'transformations')
+Import('qt_env', 'arch', 'common', 'messaging', 'visionipc', 'transformations', 'UBUNTU_FOCAL')
 
 base_libs = [common, messaging, visionipc, transformations,
              'm', 'OpenCL', 'ssl', 'crypto', 'pthread'] + qt_env["LIBS"]
@@ -8,7 +8,7 @@ base_libs = [common, messaging, visionipc, transformations,
 if arch == 'larch64':
   base_libs.append('EGL')
 
-maps = arch in ['larch64', 'aarch64', 'x86_64']
+maps = arch in ['larch64', 'aarch64', 'x86_64'] and not UBUNTU_FOCAL
 
 if arch == "Darwin":
   del base_libs[base_libs.index('OpenCL')]

selfdrive/ui/qt/maps/map_helpers.cc

@@ -143,11 +143,11 @@ std::pair<QString, QString> map_format_distance(float d, bool is_metric) {
 
   d = std::max(d, 0.0f);
   if (is_metric) {
-    return (d > 500) ? std::pair{round_distance(d / 1000), QObject::tr("km")}
-                     : std::pair{QString::number(50 * std::nearbyint(d / 50)), QObject::tr("m")};
+    return (d > 500) ? std::pair(round_distance(d / 1000), QObject::tr("km"))
+                     : std::pair(QString::number(50 * std::nearbyint(d / 50)), QObject::tr("m"));
   } else {
     float feet = d * METER_TO_FOOT;
-    return (feet > 500) ? std::pair{round_distance(d * METER_TO_MILE), QObject::tr("mi")}
-                        : std::pair{QString::number(50 * std::nearbyint(d / 50)), QObject::tr("ft")};
+    return (feet > 500) ? std::pair(round_distance(d * METER_TO_MILE), QObject::tr("mi"))
+                        : std::pair(QString::number(50 * std::nearbyint(d / 50)), QObject::tr("ft"));
   }
 }

third_party/maplibre-native-qt/x86_64/lib/libQMapLibre.so.3.0.0

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b718e4ea770105893dc41f64be36f806586a9471e2bb9d18d5ad97e906434548
-size 11000296
+oid sha256:d62e192aa7806302ed79866d9e3c51efc4bc315a6fdfe0e58e448dac5b279098
+size 11549728

tools/README.md

@@ -2,11 +2,11 @@
 
 ## System Requirements
 
-openpilot is developed and tested on **Ubuntu 20.04**, which is the primary development target aside from the [supported embedded hardware](https://github.com/commaai/openpilot#running-on-a-dedicated-device-in-a-car).
+openpilot is developed and tested on **Ubuntu 24.04**, which is the primary development target aside from the [supported embedded hardware](https://github.com/commaai/openpilot#running-on-a-dedicated-device-in-a-car).
 
 Running natively on any other system is not recommended and will require modifications. On Windows you can use WSL, and on macOS or incompatible Linux systems, it is recommended to use the dev containers.
 
-## Native setup on Ubuntu 20.04
+## Native setup on Ubuntu 24.04
 
 **1. Clone openpilot**
 
@@ -55,7 +55,7 @@ GUI apps like `ui` or `cabana` can also run inside the container by leveraging X
 
 [Windows Subsystem for Linux (WSL)](https://docs.microsoft.com/en-us/windows/wsl/about) should provide a similar experience to native Ubuntu. [WSL 2](https://docs.microsoft.com/en-us/windows/wsl/compare-versions) specifically has been reported by several users to be a seamless experience.
 
-Follow [these instructions](https://docs.microsoft.com/en-us/windows/wsl/install) to setup the WSL and install the `Ubuntu-20.04` distribution. Once your Ubuntu WSL environment is setup, follow the Linux setup instructions to finish setting up your environment. See [these instructions](https://learn.microsoft.com/en-us/windows/wsl/tutorials/gui-apps) for running GUI apps.
+Follow [these instructions](https://docs.microsoft.com/en-us/windows/wsl/install) to setup the WSL and install the `Ubuntu-24.04` distribution. Once your Ubuntu WSL environment is setup, follow the Linux setup instructions to finish setting up your environment. See [these instructions](https://learn.microsoft.com/en-us/windows/wsl/tutorials/gui-apps) for running GUI apps.
 
 **NOTE**: If you are running WSL and any GUIs are failing (segfaulting or other strange issues) even after following the steps above, you may need to enable software rendering with `LIBGL_ALWAYS_SOFTWARE=1`, e.g. `LIBGL_ALWAYS_SOFTWARE=1 selfdrive/ui/ui`.
 

tools/profiling/snapdragon/README.md

@@ -3,7 +3,7 @@ snapdragon profiler
 
 
 * download from https://developer.qualcomm.com/software/snapdragon-profiler/tools-archive (need a qc developer account)
-  * choose v2021.5 (verified working with 20.04 dev environment)
+  * choose v2021.5 (verified working with 24.04 dev environment)
 * unzip to selfdrive/debug/profiling/snapdragon/SnapdragonProfiler
 * run ```./setup-profiler.sh```
 * run ```./setup-agnos.sh```

tools/webcam/README.md

@@ -1,7 +1,7 @@
 # Run openpilot with webcam on PC
 
 What's needed:
-- Ubuntu 20.04
+- Ubuntu 24.04
 - GPU (recommended)
 - Two USB webcams, at least 720p and 78 degrees FOV (e.g. Logitech C920/C615)
 - [Car harness](https://comma.ai/shop/products/comma-car-harness) with black panda to connect to your car