Project

General

Profile

Actions

Bug #12801

closed
IA IA

GH-Runners haben Verbindungsprobleme zu apt sources

Bug #12801: GH-Runners haben Verbindungsprobleme zu apt sources

Added by Ignacio Alejandro about 2 months ago. Updated about 2 months ago.

Status:
Won't Fix
Priority:
Major
Start date:
06.03.2026
Due date:
% Done:

0%

Estimated time:
SecReporter:
Originally created on:
18.07.2025
Originally updated on:
22.07.2025
Original due date:

Description

  • Es gibt manchmals Probleme mit dem Build and Push Step von Artemis Runs in GH-self-hosten Runners
  • Vor kurz Update in Harbor, ist aber anscheinend nicht damit related
  • apt update fails mit 100

6.396 E: Failed to fetch [http://ports.ubuntu.com/ubuntu-ports/dists/noble-security/restricted/binary-arm64/Packages.gz] File has unexpected size (2197968 != 2284806). Mirror sync in progress? [IP: 91.189.91.104 80]

 

 
h2. Full Thread

 
Florian GlombikFlorian Glombik!https://a.slack-edge.com/production-standard-emoji-assets/14.0/apple-large/1f912.png!  [Jul 2nd at 3:40 PM|https://ls1tum.slack.com/archives/C1P8FF718/p1751463645373639]

Hey, I am having trouble to get a build for [General: Only display external LLM user settings when Iris or Athena is enabled #11084|https://github.com/ls1intum/Artemis/pull/11084]Is there something wrong with our GitHub Runners? [https://github.com/ls1intum/Artemis/actions/runs/16025744679/job/45213091143]The error message
1062.9 E: Failed to fetch [http://archive.ubuntu.com/ubuntu/pool/main/g/gcc-defaults/g%2b%2b_13.2.0-7ubuntu1_amd64.deb] Unable to connect to archive.ubuntu.com:80: [IP: 185.125.190.83 80] 1062.9 E: Failed to fetch [http://archive.ubuntu.com/ubuntu/pool/main/m/make-dfsg/make_4.3-4.1build2_amd64.deb] Unable to connect to archive.ubuntu.com:80: [IP: 185.125.190.83 80] 1062.9 E: Failed to fetch [http://archive.ubuntu.com/ubuntu/pool/main/d/dpkg/libdpkg-perl_1.22.6ubuntu6.1_all.deb] Unable to connect to archive.ubuntu.com:80: [IP: 185.125.190.83 80] 1062.9 E: Failed to fetch [http://archive.ubuntu.com/ubuntu/pool/main/b/bzip2/bzip2_1.0.8-5.1build0.1_amd64.deb] Unable to connect to archive.ubuntu.com:80: [IP: 185.125.190.83 80] 1062.9 E: Failed to fetch [http://archive.ubuntu.com/ubuntu/pool/main/p/patch/patch_2.7.6-7build3_amd64.deb] Unable to connect to archive.ubuntu.com:80: [IP: 185.125.190.83 80] 1062.9 E: Failed to fetch [http://archive.ubuntu.com/ubuntu/pool/main/l/lto-disabled-list/lto-disabled-list_47_all.deb] Unable to connect to archive.ubuntu.com:80: [IP: 185.125.190.83 80] 1062.9 E: Failed to fetch [http://archive.ubuntu.com/ubuntu/pool/main/d/dpkg/dpkg-dev_1.22.6ubuntu6.1_all.deb] Unable to connect to archive.ubuntu.com:80: [IP: 185.125.190.83 80] 1062.9 E: Failed to fetch [http://archive.ubuntu.com/ubuntu/pool/main/b/build-essential/build-essential_12.10ubuntu1_amd64.deb] Unable to connect to archive.ubuntu.com:80: [IP: 185.125.190.83 80] 1062.9 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing? Error: buildx failed with: ERROR: failed to build: failed to solve: process "/bin/sh -c echo "Installing build dependencies" && apt-get update && apt-get install -y --no-install-recommends python3 build-essential && apt-get clean && rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 100
 
does not sound like it would be anything that I could controlI also verified that the client and server are build locally without any problems - what can I do to get a build? I already triggered it manually again and by pushing new changes  !https://a.slack-edge.com/production-standard-emoji-assets/14.0/apple-medium/1f914@2x.png!  (edited) 
 
5 replies

Florian GlombikFlorian Glombik!https://a.slack-edge.com/production-standard-emoji-assets/14.0/apple-large/1f912.png!  [Jul 3rd at 10:55 AM|https://ls1tum.slack.com/archives/C1P8FF718/p1751532916569719?thread_ts=1751463645.373639&cid=C1P8FF718]

While for this PR I got a build by now, here is another PR where the runners are failing and it seems like it is nor related to the changes in the PR[Text exercises: Extend the learner profile with feedback preferences #10916|https://github.com/ls1intum/Artemis/pull/10916] * [Failing GitHub Runner|https://github.com/ls1intum/Artemis/actions/runs/16044596388/job/45274392047]

  • [Failing Bamboo Build|https://bamboo.ase.in.tum.de/browse/ARTEMIS-WEBAPP8892-63]
    [10:56|https://ls1tum.slack.com/archives/C1P8FF718/p1751532971950459?thread_ts=1751463645.373639&cid=C1P8FF718]
    For this PR the Bamboo build was fixed only by triggering the pipeline again, so it seems like there are problems with our pipeline (manually triggered build without any code changes that did pass [https://bamboo.ase.in.tum.de/browse/ARTEMIS-WEBAPP8892-64])

Sascha O. BacherSascha O. Bacher  [Jul 3rd at 5:45 PM|https://ls1tum.slack.com/archives/C1P8FF718/p1751557534503209?thread_ts=1751463645.373639&cid=C1P8FF718]

At a first glance, the GitHub-Runner issue seems to lie with GitHub-hosted runners since these are used for the Docker image builds. As for the Bamboo build, I am unsure what could cause this. Maybe another admin knows more.
 Also sent to the channel

Florian GlombikFlorian Glombik!https://a.slack-edge.com/production-standard-emoji-assets/14.0/apple-large/1f912.png!  [Yesterday at 2:42 PM|https://ls1tum.slack.com/archives/C1P8FF718/p1752756121795619?thread_ts=1751463645.373639&cid=C1P8FF718]

Hey, the GitHub runners are still failing randomly on the build step - are there any updates on the issue?Here is a current example: [https://github.com/ls1intum/Artemis/actions/runs/16344768580/job/46175570987]
#8 [builder 2/11] RUN echo "Installing build dependencies" && apt-get update && apt-get install -y --no-install-recommends python3 build-essential && apt-get clean && rm -rf /var/lib/apt/lists/* #8 CANCELED ------ > [runtime 2/6] RUN echo "Installing needed dependencies" && apt-get update && apt-get install -y --no-install-recommends locales graphviz wget && apt-get clean && rm -rf /var/lib/apt/lists/{}: 4.710 Get:17 [http://ports.ubuntu.com/ubuntu-ports] noble-security/main arm64 Packages [1,296 kB] 6.396 E: Failed to fetch [http://ports.ubuntu.com/ubuntu-ports/dists/noble-security/restricted/binary-arm64/Packages.gz] File has unexpected size (2197968 != 2284806). Mirror sync in progress? [IP: 91.189.91.104 80] 6.396 Hashes of expected file: 6.396 - Filesize:2284806 [weak] 6.396 - SHA256:ea476f04ba702452da8077caf91d69703670caaa63e10b180e3314ae25741823 6.396 - SHA1:a276ab713be8982fea54b6eeddc2b5cca6acb79e [weak] 6.396 - MD5Sum:6b913ad1d49302d00de289c81d41fe72 [weak] 6.396 Release file created at: Thu, 17 Jul 2025 10:54:10 +0000 6.396 E: Some index files failed to download. They have been ignored, or old ones used instead. ------ Dockerfile:99 -------------------- 98 | # Artemis: graphviz, locales 99 | >>> RUN echo "Installing needed dependencies" \ 100 | >>> && apt-get update && apt-get install -y --no-install-recommends locales graphviz wget \ 101 | >>> && apt-get clean \ 102 | >>> && rm -rf /var/lib/apt/lists/{} 103 | -------------------- ERROR: failed to build: failed to solve: process "/bin/sh -c echo "Installing needed dependencies" && apt-get update && apt-get install -y --no-install-recommends locales graphviz wget && apt-get clean && rm -rf /var/lib/apt/lists/{}" did not complete successfully: exit code: 100 Reference Check build summary support Error: buildx failed with: ERROR: failed to build: failed to solve: process "/bin/sh -c echo "Installing needed dependencies" && apt-get update && apt-get install -y --no-install-recommends locales graphviz wget && apt-get clean && rm -rf /var/lib/apt/lists/{}" did not complete successfully: exit code: 100
 

IA Updated by Ignacio Alejandro about 2 months ago Actions #1

Hi [~ge84won] ,

thanks for notifying us about the issue. I've had a look at the configuration of the network and docker network of the self-hosted runners and everything seems to be in order. Recent runs of the job also go through without problems in the APT repos.

I'd expect the issue to be related to the remote apt repositories, as they are often unstable while being updated.

My suggestion is to use a more stable repo, such as archive.ubuntu.com. An alternative would be to build logic in bash to retry after failed attempts, though the fact that apt might fail with other errors makes this suboptimal and implies that it requires logic to not run indefinitely. If you really need ubuntu ports retry logic is the best idea that I can think of though

Please let me know if the issue occurs again after applying a fix!

 

Best,

Ignacio

FG Updated by Florian Glombik about 2 months ago Actions #2

[~ge49juk] [~ge28dih] could you please have a look at this or forward the proposed solution to whoever maintains the pipelines?

IA Updated by Ignacio Alejandro about 2 months ago Actions #3

Hi [~ge25bas] ,

we've discussed the issue in our meeting and determined that it's not something that can be fixed by us. Considering that that workflow runs on GitHub's own runners, any sort of unstable behavior when connecting to apt is either unavoidable or related to the images used in the workflow.

We'd suggest verifying the image versions and architectures. 

 

Best,

Ignacio

Actions

Also available in: PDF Atom