1.2.0 release notes and a few final tweaks and cleanup.

6 files changed, 90 insertions, 46 deletions

diff --git a/RELEASE-NOTES.md b/RELEASE-NOTES.md
index 42a2aaa4..d0ad47d8 100644
--- a/RELEASE-NOTES.md
+++ b/RELEASE-NOTES.md
@@ -1,32 +1,30 @@
 ZeroTier Release Notes
 ======
 
-*As of 1.2.0 this will serve as a detailed changelog, which we've needed for a long time.*
+# 2017-03-14 -- Version 1.2.0
 
-# 2017-03-13 -- Version 1.2.0
+Version 1.2.0 is a major milestone release representing almost nine months of work. It includes our rules engine for distributed network packet filtering and security monitoring, federated roots, and many other architectural and UI improvements and bug fixes.
 
-Version 1.2.0 is a major milestone release and introduces a large number of new capabilities to the ZeroTier core network hypervisor. It also includes some security tightening, major UI improvements for Windows and Macintosh platforms, and a number of bug fixes and platform issue workarounds.
-
-## Features in 1.2.0
+## New Features in 1.2.0
 
 ### The ZeroTier Rules Engine
 
 The largest new feature in 1.2.0, and the product of many months of work, is our advanced network rules engine. With this release we achieve traffic control, security monitoring, and micro-segmentation capability on par with many enterprise SDN solutions designed for use in advanced data centers and corporate networks.
 
-Rules allow you to filter packets on your network and vector traffic to security observers (e.g. a node running Snort). Security observation can be performed in-band using REDIRECT or out of band using TEE, and for tha latter it can be done for headers only, for select traffic, or probabilistically to reduce overhead on large distributed networks.
+Rules allow you to filter packets on your network and vector traffic to security observers. Security observation can be performed in-band using REDIRECT or out of band using TEE.
 
 Tags and capabilites provide advanced methods for implementing fine grained permission structures and micro-segmentation schemes without bloating the size and complexity of your rules table.
 
-See our manual for more information.
+See the [rules engine announcement blog post](https://www.zerotier.com/blog/?p=927) for an in-depth discussion of theory and implementation. The [manual](https://www.zerotier.com/manual.shtml) contains detailed information on rule, tag, and capability use, and the `rule-compiler/` subfolder of the ZeroTier source tree contains a JavaScript function to compile rules in our human-readable rule definition language into rules suitable for import into a network controller. (ZeroTier Central uses this same script to compile rules on [my.zerotier.com](https://my.zerotier.com/).)
 
 ### Root Server Federation
 
-It's now possible to create your own root servers and add them to the root server pool on your nodes. This is done by creating what's called a "moon," which is a signed enumeration of root servers and their stable points on the network. Refer to the manual for more details on how to do this and how it works.
+It's now possible to create your own root servers and add them to the root server pool on your nodes. This is done by creating what's called a "moon," which is a signed enumeration of root servers and their stable points on the network. Refer to the [manual](https://www.zerotier.com/manual.shtml) for instructions.
 
 Federated roots achieve a number of things:
 
  * You can deploy your own infrastructure to reduce dependency on ours.
- * You can deploy them *inside your LAN* to ensure that network connectivity inside your facility still works if the Internet goes down. This is the first step toward making ZeroTier viable as an in-house SDN solution.
+ * You can deploy roots *inside your LAN* to ensure that network connectivity inside your facility still works if the Internet goes down. This is the first step toward making ZeroTier viable as an in-house SDN solution.
  * Roots can be deployed inside national boundaries for countries with data residency laws or "great firewalls." (As of 1.2.0 there is still no way to force all traffic to use these roots, but that will be easy to do in a later version.)
  * Last but not least this makes ZeroTier somewhat less centralized by eliminating any hard dependency on ZeroTier, Inc.'s infrastructure.
 
@@ -50,23 +48,28 @@ A good software update system for Windows and Mac clients has been a missing fea
 
 We've greatly improved this mechanism in 1.2.0. Not only does it now do a better job of actually invoking the update, but it also transfers updates in-band using the ZeroTier protocol. This means it can work in environments that do not allows http/https traffic or that force it through proxies. There's also now an update channel setting: `beta` or `release` (the default).
 
-As before software updates are authenticated in two ways:
+Software updates are authenticated three ways:
+
+ 1. ZeroTier's own signing key is used to sign all updates and this signature is checked prior to installation. ZeroTier, Inc.'s signatures are performed on an air-gapped machine.
 
- 1. ZeroTier's own signing key is used to sign all updates and this signature is checked prior to installation. Our signatures are done on an air-gapped machine.
+ 2. Updates for Mac and Windows are signed using Apple and Microsoft (DigiCert EV) keys and will not install unless these signatures are also valid.
 
- 2. Updates for Mac and Windows are signed using Apple and Microsoft (DigiCert) keys and will not install unless these signatures are also valid.
+ 3. The new in-band update mechanism also authenticates the source of the update via ZeroTier's built-in security features. This provides transport security, while 1 and 2 provide security of the update at rest.
 
-Version 1.2.0's in-band mechanism effectively adds a third way: updates are fetched in-band from a designated ZeroTier node, thus authenticating the source using ZeroTier's built-in encryption and authentication mechanisms.
+Updates are now configurable via `local.conf`. There are three options: `disable`, `download`, and `apply`. The third (apply) is the default for official builds on Windows and Mac, making updates happen silently and automatically as they do for popular browsers like Chrome and Firefox. Updates are disabled by default on Linux and other Unix-type systems as these are typically updated through package managers.
 
-Updates are now configurable via `local.conf`. There are three options: `disable`, `download`, and `apply`. The third is the default for official builds on Windows and Mac, making updates happen silently and automatically as they do for popular browsers like Chrome and Firefox. For managed enterprise deployments IT people could ship a local.conf that disables updates and instead push updates via their management capabilities. Updates are disabled on Linux and other Unix-type platforms as these get updates through package repositories.
+### Path Link Quality Awareness
 
-### Path Quality Monitoring (QoS and SD-WAN phase one)
+Version 1.2.0 is now aware of the link quality of direct paths with other 1.2.0 nodes. This information isn't used yet but is visible through the JSON API. (Quality always shows as 100% with pre-1.2.0 nodes.) Quality is measured passively with no additional overhead using a counter based packet loss detection algorithm.
 
-Version 1.2.0 is now aware of the link quality of direct paths with other 1.2.0 nodes. This information isn't used yet but is visible through the JSON API. (Quality always shows as 100% with pre-1.2.0 nodes.)
+This information is visible from the command line via `listpeers`:
 
-Link quality monitoring is a precursor to intelligent multi-path and QoS support, which will in future versions bring us to feature parity with SD-WAN products like Cisco iWAN.
+    200 listpeers XXXXXXXXXX 199.XXX.XXX.XXX/9993;10574;15250;1.00 48 1.2.0 LEAF
+    200 listpeers XXXXXXXXXX 195.XXX.XXX.XXX/45584;467;7608;0.44 290 1.2.0 LEAF
 
-"Connect all the things!"
+The first peer's path is at 100% (1.00), while the second peer's path is suffering quite a bit of packet loss (0.44).
+
+Link quality awareness is a precursor to intelligent multi-path and QoS support, which will in future versions bring us to feature parity with SD-WAN products like Cisco iWAN.
 
 ### Security Improvements
 
@@ -78,15 +81,42 @@ Revocations propagate using a "rumor mill" peer to peer algorithm. This means th
 
 ### Windows and Macintosh UI Improvements (ZeroTier One)
 
-The Mac has a whole new UI built natively in Objective-C. It provides a pulldown similar in appearance and operation to the Mac WiFi task bar menu. The Windows UI has also been improved and now provides a task bar icon that can be right-clicked to manage networks. Both now expose managed route and IP permissions, allowing nodes to easily opt in to full tunnel operation if you have a router configured on your network.
+The Mac has a whole new UI built natively in Objective-C. It provides a pulldown similar in appearance and operation to the Mac WiFi task bar menu.
+
+The Windows UI has also been improved and now provides a task bar icon that can be right-clicked to manage networks. Both now expose managed route and IP permissions, allowing nodes to easily opt in to full tunnel operation if you have a router configured on your network.
+
+### Ad-Hoc Networks
+
+A special kind of public network called an ad-hoc network may be accessed by joining a network ID with the format:
+
+    ffSSSSEEEE000000
+    | |   |   |
+    | |   |   Reserved for future use, must be 0
+    | |   End of port range (hex)
+    | Start of port range (hex)
+    Reserved ZeroTier address prefix indicating a controller-less network
+
+Ad-hoc networks are public (no access control) networks that have no network controller. Instead their configuration and other credentials are generated locally. Ad-hoc networks permit only IPv6 UDP and TCP unicast traffic (no multicast or broadcast) using 6plane format NDP-emulated IPv6 addresses. In addition an ad-hoc network ID encodes an IP port range. UDP packets and TCP SYN (connection open) packets are only allowed to desintation ports within the encoded range.
+
+For example `ff00160016000000` is an ad-hoc network allowing only SSH, while `ff0000ffff000000` is an ad-hoc network allowing any UDP or TCP port.
+
+Keep in mind that these networks are public and anyone in the entire world can join them. Care must be taken to avoid exposing vulnerable services or sharing unwanted files or other resources.
+
+### Network Controller (Partial) Rewrite
+
+The network controller has been largely rewritten to use a simple in-filesystem JSON data store in place of SQLite, and it is now included by default in all Windows, Mac, Linux, and BSD builds. This means any desktop or server node running ZeroTier One can now be a controller with no recompilation needed.
+
+If you have data in an old SQLite3 controller we've included a NodeJS script in `controller/migrate-sqlite` to migrate data to the new format. If you don't migrate, members will start getting `NOT_FOUND` when they attempt to query for updates.
 
 ## Major Bug Fixes in 1.2.0
 
- * **The Windows HyperV 100% CPU bug**
-   * This long-running problem turns out to have been an issue with Windows itself, but one we were triggering by placing invalid data into the Windows registry. Microsoft is aware of the issue but we've also fixed the triggering problem on our side. ZeroTier should now co-exist quite well with HyperV and should now be able to be bridged with a HyperV virtual switch.
- * **Segmenation Faults on musl-libc based Linux Systems**
-   * Alpine Linux and some embedded Linux systems that use musl libc (a minimal libc) experienced segmentation faults. These were due to a smaller default stack size. A work-around that sets the stack size for new threads has been added.
- * **Windows Firewall Blocks Local JSON API**
-   * On some Windows systems the firewall likes to block 127.0.0.1:9993 for mysterious reasons. This is now fixed in the installer via the addition of another firewall exemption rule.
- * **UI Crash on Embedded Windows Due to Missing Fonts**
-   * The MSI installer now ships fonts and will install them if they are not present, so this should be fixed.
+ * **The Windows HyperV 100% CPU bug is FINALLY DEAD**: This long-running problem turns out to have been an issue with Windows itself, but one we were triggering by placing invalid data into the Windows registry. Microsoft is aware of the issue but we've also fixed the triggering problem on our side. ZeroTier should now co-exist quite well with HyperV and should now be able to be bridged with a HyperV virtual switch.
+ * **Segmenation faults on musl-libc based Linux systems**: Alpine Linux and some embedded Linux systems that use musl libc (a minimal libc) experienced segmentation faults. These were due to a smaller default stack size. A work-around that sets the stack size for new threads has been added.
+ * **Windows firewall blocks local JSON API**: On some Windows systems the firewall likes to block 127.0.0.1:9993 for mysterious reasons. This is now fixed in the installer via the addition of another firewall exemption rule.
+ * **UI crash on embedded Windows due to missing fonts**: The MSI installer now ships fonts and will install them if they are not present, so this should be fixed.
+
+## Other Improvements in 1.2.0
+
+ * **Improved dead path detection**: ZeroTier is now more aggressive about expiring paths that do not seem to be active. If a path seems marginal it is re-confirmed before re-use.
+ * **Minor performance improvements**: We've reduced unnecessary memcpy's and made a few other performance improvements in the core.
+ * **Linux static binaries**: For our official packages (the ones in the download.zerotier.com apt and yum repositories) we now build Linux binaries with static linking. Hopefully this will stop all the bug reports relating to library inconsistencies, as well as allowing our deb packages to run on a wider variety of Debian-based distributions. (There are far too many of these to support officially!) The overhead for this is very small, especially since we built our static versions against musl-libc. Distribution maintainers are of course free to build dynamically linked versions for inclusion into distributions; this only affects our official binaries.
diff --git a/rule-compiler/README.md b/rule-compiler/README.md
index e3aa2615..1ceeb713 100644
--- a/rule-compiler/README.md
+++ b/rule-compiler/README.md
@@ -3,3 +3,6 @@ ZeroTier Rules Compiler
 
 This script converts ZeroTier rules in human-readable format into rules suitable for import into a ZeroTier network controller. It's the script that is used in the rules editor on [ZeroTier Central](https://my.zerotier.com/).
 
+A command line interface is included that may be invoked as: `node cli.js <rules script>`.
+
+See the [manual](https://www.zerotier.com/manual.shtml) for information about the rules engine and rules script syntax.
diff --git a/rule-compiler/cli.js b/rule-compiler/cli.js
index c4a3b291..a0ff5197 100644
--- a/rule-compiler/cli.js
+++ b/rule-compiler/cli.js
@@ -1,7 +1,6 @@
 'use strict';
 
 var fs = require('fs');
-
 var RuleCompiler = require('./rule-compiler.js');
 
 if (process.argv.length < 3) {
@@ -9,21 +8,39 @@ if (process.argv.length < 3) {
 	process.exit(1);
 }
 
-var src = fs.readFileSync(process.argv[2]).toString();
-
 var rules = [];
 var caps = {};
 var tags = {};
-var err = RuleCompiler.compile(src,rules,caps,tags);
+var err = RuleCompiler.compile(fs.readFileSync(process.argv[2]).toString(),rules,caps,tags);
 
 if (err) {
-	console.log('ERROR parsing '+process.argv[2]+' line '+err[0]+' column '+err[1]+': '+err[2]);
+	console.error('ERROR parsing '+process.argv[2]+' line '+err[0]+' column '+err[1]+': '+err[2]);
 	process.exit(1);
 } else {
+	let capsArray = [];
+	let capabilitiesByName = {};
+	for(let n in caps) {
+		capsArray.push(caps[n]);
+		capabilitiesByName[n] = caps[n].id;
+	}
+	let tagsArray = [];
+	for(let n in tags) {
+		let t = tags[n];
+		tagsArray.push({
+			'id': t.id,
+			'default': t['default']||null
+		});
+	}
+
 	console.log(JSON.stringify({
-		rules: rules,
-		caps: caps,
-		tags: tags
-	},null,2));
+		config: {
+			rules: rules,
+			capabilities: capsArray,
+			tags: tagsArray
+		},
+		capabilitiesByName: capabilitiesByName,
+		tagsByName: tags
+	},null,1));
+
 	process.exit(0);
 }
diff --git a/rule-compiler/package.json b/rule-compiler/package.json
index 451295a4..e12d3759 100644
--- a/rule-compiler/package.json
+++ b/rule-compiler/package.json
@@ -1,6 +1,6 @@
 {
   "name": "zerotier-rule-compiler",
-  "version": "1.1.17-3",
+  "version": "1.2.0-1",
   "description": "ZeroTier Rule Script Compiler",
   "main": "cli.js",
   "scripts": {
diff --git a/service/OneService.cpp b/service/OneService.cpp
index 129c0499..4a2102f1 100644
--- a/service/OneService.cpp
+++ b/service/OneService.cpp
@@ -745,14 +745,6 @@ public:
 			for(int i=0;i<3;++i)
 				_portsBE[i] = Utils::hton((uint16_t)_ports[i]);
 
-			// Check for legacy controller.db and terminate if present to prevent nasty surprises for DIY controller folks
-			if (OSUtils::fileExists((_homePath + ZT_PATH_SEPARATOR_S "controller.db").c_str())) {
-				Mutex::Lock _l(_termReason_m);
-				_termReason = ONE_UNRECOVERABLE_ERROR;
-				_fatalErrorMessage = "controller.db is present in our home path! run migrate-sqlite to migrate to new controller.d format.";
-				return _termReason;
-			}
-
 			_controller = new EmbeddedNetworkController(_node,(_homePath + ZT_PATH_SEPARATOR_S ZT_CONTROLLER_DB_PATH).c_str());
 			_node->setNetconfMaster((void *)_controller);
 
diff --git a/service/README.md b/service/README.md
index bdf713c1..f5223f2d 100644
--- a/service/README.md
+++ b/service/README.md
@@ -27,6 +27,7 @@ Settings available in `local.conf` (this is not valid JSON, and JSON does not al
 		"primaryPort": 0-65535, /* If set, override default port of 9993 and any command line port */
 		"portMappingEnabled": true|false, /* If true (the default), try to use uPnP or NAT-PMP to map ports */
 		"softwareUpdate": "apply"|"download"|"disable", /* Automatically apply updates, just download, or disable built-in software updates */
+		"softwareUpdateChannel": "release"|"beta", /* Software update channel */
 		"softwareUpdateDist": true|false, /* If true, distribute software updates (only really useful to ZeroTier, Inc. itself, default is false) */
 		"interfacePrefixBlacklist": [ "XXX",... ], /* Array of interface name prefixes (e.g. eth for eth#) to blacklist for ZT traffic */
 		"allowManagementFrom": "NETWORK/bits"|null /* If non-NULL, allow JSON/HTTP management from this IP network. Default is 127.0.0.1 only. */
@@ -57,7 +58,8 @@ An example `local.conf`:
 		}
 	},
 	"settings": {
-		"relayPolicy": "ALWAYS"
+		"softwareUpdate": "apply",
+		"softwraeUpdateChannel": "release"
 	}
 }
 ```