build-tools: debian: image creation fails intermittently
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Fix Released
|
Medium
|
Unassigned |
Bug Description
Brief Description
-----------------
Debian build tools sometimes fail when building the ISO image. The problem seems to be related to the aptly POD, or k8s networking.
apt-get inside the LAT fails to download packages from repomgr pod with "Connection reset by peer" or "Broken pipe" errors. This happens intermittently with different packages. This error was observed on vanilla k8s, not Minikube.
Severity
--------
Major
Steps to Reproduce
------------------
Set up a debian build environment in vanilla k8s (not minikube). Build packages as usual. Build the image.
Expected Behavior
------------------
Image build succeeds
Actual Behavior
----------------
Image build fails intermittently.
Reproducibility
---------------
Intermittent
System Configuration
-------
N/A
Branch/Pull Time/Commit
-------
master/2021-03-04
Last Pass
---------
master/2021-03-03
Timestamp/Logs
--------------
08:12:39 appsdk - DEBUG: E: Failed to fetch http://
Test Activity
-------------
N/A
Workaround
----------
Try the build again, or build in minikube.
tags: | added: stx.7.0 stx.build stx.debian |
Changed in starlingx: | |
importance: | Undecided → Medium |
status: | New → Triaged |
I can reproduce this problem vanilla k8s approx 30% of the time.
I haven't specifically tried to reproduce it in minikube, but nobody has complained about minikube so far.
I tried setting "net.netfilter. nf_conntrack_ tcp_be_ liberal= 1" kernel option on the host as described here: https:/ /technology. lastminute. com/chasing- k8s-connection- reset-issue/ -- it doesn't seem to help.