hexdump.h: fix arithmetic error.

When I modified the hexdumper to help debug MokListRT mirroring not
working because of PcdMaxVolatileVariableSize being tiny, I
inadvertently added something that is effectively:

hexdump(..., char *buf, ..., int position)
{
	unsigned long begin = (position % 16);
	unsigned long i;
	...
	for (i = 0; i < begin; i++) {
		...
	}
	...
}

Unfortunately, in c if 0x8 is set in position, that means begin is
0xfffffffffffff8, because signed integer math is horrifying:

include/hexdump.h:99:vhexdumpf() &data[offset]:0x9E77E6BC size-offset:0x14
include/hexdump.h:15:prepare_hex() position:0x9E77E6BC
include/hexdump.h:17:prepare_hex() before:0xFFFFFFFFFFFFFFFC size:0x14
include/hexdump.h:19:prepare_hex() before:0xFFFFFFFFFFFFFFFC after:0x0
include/hexdump.h:21:prepare_hex() buf:0x000000009E77E2BC offset:0 &buf[offset]:0x000000009E77E2BC

Woops.

This could further have been prevented in /some/ cases by simply not
preparing the hexdump buffer when "verbose" is disabled.

This patch makes "pos" be unsigned in all cases, and also checks for
verbose in vhexdumpf() and simply returns if it is 0.

Signed-off-by: Peter Jones <pjones@redhat.com>

1 files changed, 5 insertions, 2 deletions

diff --git a/include/hexdump.h b/include/hexdump.h
index f3f3ac28..b2968cd4 100644
--- a/include/hexdump.h
+++ b/include/hexdump.h
@@ -4,7 +4,7 @@
 #include <stdint.h>
 
 static inline unsigned long UNUSED
-prepare_hex(const void *data, size_t size, char *buf, int position)
+prepare_hex(const void *data, size_t size, char *buf, unsigned int position)
 {
 	char hexchars[] = "0123456789abcdef";
 	int offset = 0;
@@ -48,7 +48,7 @@ prepare_hex(const void *data, size_t size, char *buf, int position)
 #define isprint(c) ((c) >= 0x20 && (c) <= 0x7e)
 
 static inline void UNUSED
-prepare_text(const void *data, size_t size, char *buf, int position)
+prepare_text(const void *data, size_t size, char *buf, unsigned int position)
 {
 	int offset = 0;
 	unsigned long i;
@@ -84,6 +84,9 @@ vhexdumpf(const char *file, int line, const char *func, const CHAR16 * const fmt
 	unsigned long display_offset = at;
 	unsigned long offset = 0;
 
+	if (verbose == 0)
+		return;
+
 	while (offset < size) {
 		char hexbuf[49];
 		char txtbuf[19];