From c53a94932583ad8030594dd021e4a035ffcc872f Mon Sep 17 00:00:00 2001
From: Lyndon Brown <jnqnfe@gmail.com>
Date: Mon, 10 Feb 2020 18:49:34 +0000
Subject: firmware: fix possible duplication in firmware package lists
 (inefficiency)

where multiple archive areas are used, the code here on each loop is:
  1) fetching the archive area contents file (compressed)
  2) **appending** the output to that of the previous loop
  3) searching the file for firmware references, adding to the list

since it appends rather than replaces, entries found in each loop get
re-added on each subsequent loop, resulting in duplication in the
resulting list

below I evaluate the possible solutions to explain why I chose the one
I chose, however the reader should not waste too much time worrying about
whether one of the other solution would have actually been better because
things are changed significantly in further commits shortly!

possible solutions:
  a) switching to output (>) rather than append (>>), but this might fail
     against an existing file
  b) removing the file on each loop, but this will complicate any future
     caching improvements that might be made here (currently the files are
     always deleted and thus downloaded fresh)
  c) allow the appending, evaluating the complete file after the loop

solution C warrants consideration of disk space consumption; currently the
compressed 'main' archive (for sid on amd64) expands to 592.3 MB (feb-2020),
'contrib' is 3.1 MB, and 'non-free' is 18.5 MB.

solution C was chosen here; the difference of accumulated file size vs.
max-single was minor enough to not be of particular concern (~613 vs.
~592 MB).

Gbp-Dch: Short
Closes: #952906
---
 scripts/build/chroot_firmware            | 8 ++++----
 scripts/build/installer_debian-installer | 8 ++++----
 2 files changed, 8 insertions(+), 8 deletions(-)

(limited to 'scripts/build')

diff --git a/scripts/build/chroot_firmware b/scripts/build/chroot_firmware
index 4cfe37e48..1070ccbae 100755
--- a/scripts/build/chroot_firmware
+++ b/scripts/build/chroot_firmware
@@ -68,10 +68,10 @@ do
 	CONTENTS_URL="${LB_PARENT_MIRROR_CHROOT}/dists/${LB_PARENT_DISTRIBUTION_CHROOT}/${_PARENT_ARCHIVE_AREA}/Contents-${LB_ARCHITECTURES}.gz"
 
 	wget ${WGET_OPTIONS} "${CONTENTS_URL}" -O - | gunzip -c >> "${CONTENTS_FILE}"
-
-	FIRMWARE_PACKAGES="${FIRMWARE_PACKAGES} $(awk '/^lib\/firmware/ { print $2 }' "${CONTENTS_FILE}" | sort -u)"
 done
 
+FIRMWARE_PACKAGES="${FIRMWARE_PACKAGES} $(awk '/^lib\/firmware/ { print $2 }' "${CONTENTS_FILE}" | sort -u)"
+
 if echo ${LB_PARENT_ARCHIVE_AREAS} | grep -qs "non-free"
 then
 	# Manually add firmware-linux/non-free meta package
@@ -102,9 +102,9 @@ then
 		CONTENTS_URL="${LB_MIRROR_CHROOT}/dists/${LB_DISTRIBUTION_CHROOT}/${_ARCHIVE_AREA}/Contents-${LB_ARCHITECTURES}.gz"
 
 		wget ${WGET_OPTIONS} "${CONTENTS_URL}" -O - | gunzip -c >> "${CONTENTS_FILE}"
-
-		FIRMWARE_PACKAGES="${FIRMWARE_PACKAGES} $(awk '/^lib\/firmware/ { print $2 }' "${CONTENTS_FILE}" | sort -u)"
 	done
+
+	FIRMWARE_PACKAGES="${FIRMWARE_PACKAGES} $(awk '/^lib\/firmware/ { print $2 }' "${CONTENTS_FILE}" | sort -u)"
 fi
 
 # Drop section and keep package names only
diff --git a/scripts/build/installer_debian-installer b/scripts/build/installer_debian-installer
index 4d0b56ffc..b4802b3f9 100755
--- a/scripts/build/installer_debian-installer
+++ b/scripts/build/installer_debian-installer
@@ -356,10 +356,10 @@ then
 			CONTENTS_URL="${LB_PARENT_MIRROR_CHROOT}/dists/${LB_PARENT_DISTRIBUTION_CHROOT}/${_PARENT_ARCHIVE_AREA}/Contents-${LB_ARCHITECTURES}.gz"
 
 			wget ${WGET_OPTIONS} "${CONTENTS_URL}" -O - | gunzip -c >> "${CONTENTS_FILE}"
-
-			FIRMWARE_PACKAGES="${FIRMWARE_PACKAGES} $(awk '/^lib\/firmware/ { print $2 }' "${CONTENTS_FILE}" | sort -u)"
 		done
 
+		FIRMWARE_PACKAGES="${FIRMWARE_PACKAGES} $(awk '/^lib\/firmware/ { print $2 }' "${CONTENTS_FILE}" | sort -u)"
+
 		if echo ${LB_PARENT_ARCHIVE_AREAS} | grep -qs "non-free"
 		then
 			# Manually add firmware-linux/non-free meta package
@@ -390,9 +390,9 @@ then
 				CONTENTS_URL="${LB_MIRROR_CHROOT}/dists/${LB_DISTRIBUTION_CHROOT}/${_ARCHIVE_AREA}/Contents-${LB_ARCHITECTURES}.gz"
 
 				wget ${WGET_OPTIONS} "${CONTENTS_URL}" -O - | gunzip -c >> "${CONTENTS_FILE}"
-
-				FIRMWARE_PACKAGES="${FIRMWARE_PACKAGES} $(awk '/^lib\/firmware/ { print $2 }' "${CONTENTS_FILE}" | sort -u)"
 			done
+
+			FIRMWARE_PACKAGES="${FIRMWARE_PACKAGES} $(awk '/^lib\/firmware/ { print $2 }' "${CONTENTS_FILE}" | sort -u)"
 		fi
 
 		# Drop section and keep package names only
-- 
cgit v1.2.3