remote: T8829: fall back to GET when HEAD is not supported

HttpC.download() always probed remote URLs with HEAD before GET to
discover redirects and Content-Length. Some APIs (e.g. AbuseIPDB)
reject HEAD with 405 Method Not Allowed, causing firewall remote-group
downloads to fail and leave empty cached list files.

Treat HEAD 405/501 as unsupported and proceed with GET using the
original URL. When HEAD does not provide Content-Length, read it from
the GET response headers instead. Validate available storage after
determining file size and before opening the destination file.

Log sanitized download errors in vyos-domain-resolver when a remote-
group list-file fetch fails. Add unit tests with mock servers that
return 405 or 501 on HEAD and 200 on GET.

1 files changed, 14 insertions, 2 deletions

diff --git a/src/services/vyos-domain-resolver b/src/services/vyos-domain-resolver
index 65b2a476b..60fcb7df4 100755
--- a/src/services/vyos-domain-resolver
+++ b/src/services/vyos-domain-resolver
@@ -17,6 +17,8 @@ import json
 import time
 import logging
 import os
+from urllib.parse import urlsplit
+from urllib.parse import urlunsplit
 
 from vyos.configdict import dict_merge
 from vyos.configquery import ConfigTreeQuery
@@ -155,8 +157,18 @@ def update_remote_group(config):
             # Attempt to download file, use cached version if download fails
             try:
                 download(list_file, remote_config['url'], raise_error=True)
-            except:
-                logger.error(f'Failed to download list-file for {set_name} remote group')
+            except Exception as err:
+                url = urlsplit(remote_config['url'])
+                _, _, netloc = url.netloc.rpartition('@')
+                redacted_url = urlunsplit(url._replace(netloc=netloc, query='', fragment=''))
+                err_msg = str(err).replace(remote_config['url'], redacted_url)
+                logger.error(
+                    'Failed to download list-file for %s remote group: %s: %s',
+                    set_name,
+                    err.__class__.__name__,
+                    err_msg,
+                )
+                logger.debug('Download failure details for %s', set_name, exc_info=True)
                 logger.info(f'Using cached list-file for {set_name} remote group')
 
             # Read list file