(mirror of https://github.com/marekm72/vyos-cloud-init.git) https://git.amelek.net/marekm72/vyos-cloud-init.git/atom?h=20.3 2020-08-20T19:04:03+00:00 2020-08-20T19:04:03+00:00 Moustafa Moustafa momousta@microsoft.com 2020-08-20T19:04:03+00:00 urn:sha1:d941c7f7846e0216873384044d20f4b6723697c5 Push the cloud-init.log file (Up to 500KB at once) to the KVP before reporting ready to the Azure platform. Based on the analysis done on a large sample of cloud-init.log files, Here's the statistics collected on the log file size: P50 P90 P95 P99 P99.9 P99.99 137K 423K 537K 3.5MB 6MB 16MB This change limits the size of cloud-init.log file data that gets dumped to KVP to 500KB. So for ~95% of the cases, the whole log file will be dumped and for the remaining ~5%, we will get the last 500KB of the cloud-init.log file. To asses the performance of the 500KB limit, 250 VM were deployed with a 500KB cloud-init.log file and the time taken to compress, encode and dump the entries to KVP was measured. Here's the time in milliseconds percentiles: P50 P99 P999 75.705 232.701 1169.636 Another 250 VMs were deployed with this logic dumping their normal cloud-init.log file to KVP, the same timing was measured as above. Here's the time in milliseconds percentiles: P50 P99 P999 1.88 5.277 6.992 Added excluded_handlers to the report_event function to be able to opt-out from reporting the events of the compressed cloud-init.log file to the cloud-init.log file. The KVP break_down logic had a bug, where it will reuse the same key for all the split chunks of KVP which results in overwriting the split KVPs by the last one when consumed by Hyper-V. I added the split chunk index as a differentiator to the KVP key. The Hyper-V consumes the KVPs from the KVP file as chunks whose key is 512KB and value is 2048KB but the Azure platform expects the value to be 1024KB, thus I introduced the Azure value limit. 2020-07-15T14:26:12+00:00 Daniel Watkins oddbloke@ubuntu.com 2020-07-15T14:26:12+00:00 urn:sha1:4fe576516d65feda17ba78e9265a8e494a195e7b * cloudinit: remove global disable of pylint W0107 and fix errors This includes removing a test class which contained no tests but wasn't detected as empty because of an errant pass statement. * .pylintrc: update disable comment to match arguments 2020-06-10T14:39:29+00:00 Joshua Powers josh.powers@canonical.com 2020-06-10T14:39:29+00:00 urn:sha1:f3bd42659efeed4b092ffcdfd5df7f24813f2d3e 2020-06-02T16:06:07+00:00 Joshua Powers josh.powers@canonical.com 2020-06-02T16:06:07+00:00 urn:sha1:d53921ea3396c8301c65cad3abf04b4542d4b7a0 This fixes issues with closing brackets not matching the opening bracket's line and continuation line under-idented for hanging indent. 2020-01-21T22:15:30+00:00 Daniel Watkins oddbloke@ubuntu.com 2020-01-21T22:15:30+00:00 urn:sha1:8c4fd886931abcf2cc8627a47463907d655b35c3 * url_helper: drop six * url_helper: sort imports * log: drop six * log: sort imports * handlers/__init__: drop six * handlers/__init__: sort imports * user_data: drop six * user_data: sort imports * sources/__init__: drop six * sources/__init__: sort imports * DataSourceOVF: drop six * DataSourceOVF: sort imports * sources/helpers/openstack: drop six * sources/helpers/openstack: sort imports * mergers/m_str: drop six This also allowed simplification of the logic, as we will never encounter a non-string text type. * type_utils: drop six * mergers/m_dict: drop six * mergers/m_list: drop six * cmd/query: drop six * mergers/__init__: drop six * net/cmdline: drop six * reporting/handlers: drop six * reporting/handlers: sort imports 2019-11-08T21:02:07+00:00 momousta momousta 2019-11-08T21:02:07+00:00 urn:sha1:92cc71dec8da73bf43d345418b73db1a32a6b8b9 The KVPs currently being emitted to the .kvp_pool file can have duplicate keys which is wrong since these keys should be unique. The situation can occur if for example one azure function called twice or more and this function is reporting telemetry through the use of KVPs. Any KVP consumer can get confused by the duplicate keys and a race condition can and have occurred. 2019-04-29T20:22:16+00:00 Anh Vo anhvo@microsoft.com 2019-04-29T20:22:16+00:00 urn:sha1:86674f013dfcea3c075ab41373ffb475881066f6  + Truncate KVP Pool file to prevent stale entries from being processed by the Hyper-V KVP reporter.  + Drop filtering of KVPs as it is no longer needed.  + Batch appending of existing KVP entries. 2018-08-31T21:47:18+00:00 Ryan Harper ryan.harper@canonical.com 2018-08-31T21:47:18+00:00 urn:sha1:43e51a04515686a15c410d1a16dd5ff06fd1afd4 Switch the implementation to a daemon thread which uses a blocking get from the Queue. No additional locking or flag checking is needed since the Queue itself handles acquiring the lock as needed. cloud-init only has a single producer (the main thread calling publish) and the consumer will read all events in the queue and write them out. Using the daemon mode of the thread handles flushing the queue on main exit in python3; in python2.7 we handle the EOFError that results when the publish thread calls to get() fails indicating the main thread has exited. The result is that the handler is no longer spawing a thread on each publish event but rather creates a single thread when we start up the reporter and we remove any additional use of separate locks and flags as we only have a single Queue object and we're only calling queue.put() from main thread and queue.get() from consuming thread. 2018-08-24T22:25:37+00:00 Andy Liu andyliuliming@outlook.com 2018-08-24T22:25:37+00:00 urn:sha1:2320c3de2712c2f320b0d8af4aa129219cc2ad04 Linux guests can provide information to Hyper-V hosts via KVP. KVP allows the guests to provide any string key-value-pairs back to the host's registry. On linux, kvp communication pools are presented as pool files in /var/lib/hyperv/.kvp_pool_#. The following reporting configuration can enable this kvp reporting in addition to default logging if the pool files exist: reporting:     logging:         type: log     telemetry:         type: hyperv 2018-04-20T03:30:08+00:00 Scott Moser smoser@ubuntu.com 2018-04-20T03:30:08+00:00 urn:sha1:1081962eacf2814fea6f4fa3255c530de14e4a24 This enables warnings produced by pylint for unused variables (W0612), and fixes the existing errors.