From 0d37b7b66522d6a8415be8b0361df398fa1f5d6e Mon Sep 17 00:00:00 2001 From: Joseph Henry Date: Wed, 17 Aug 2016 15:12:48 -0700 Subject: zt-kubernetes tutorial outline --- doc/ext/ztkube.md | 96 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 96 insertions(+) create mode 100644 doc/ext/ztkube.md (limited to 'doc') diff --git a/doc/ext/ztkube.md b/doc/ext/ztkube.md new file mode 100644 index 00000000..483a0ff6 --- /dev/null +++ b/doc/ext/ztkube.md @@ -0,0 +1,96 @@ +Kubernetes + ZeroTier +==== + +A self-authorizing Kubernetes deployment on a ZeroTier private network. + +This is a quick tutorial for setting up a Kubernetes deployment which can self-authorize each new replica onto your private ZeroTier network with no additional configuration needed when you scale. The Kubernetes-specific instructions and content is based on the [hellonode](http://kubernetes.io/docs/hellonode/) tutorial. + + + +## Preliminary tasks + +**Step 1: Go to [my.zerotier.com](https://www.my.zerotier.com) and generate an API key. This key will be used by ZeroTier to automatically authorize new instances of your VMs to join your deployment network during replication. + +**Step 2: Create a new `private` network. Take note of the network ID: `nwid` + +**Step 3: Follow the instructions from the [hellonode](ttp://kubernetes.io/docs/hellonode/) tutorial and set up your development system (install Google Cloud SDK). + + + + +## Construct docker container + +Step 4: Create necessary files for inclusion in Dockerfile + - `mkdir ztkube` + +### Add the following files to the `ztkube` directory. These files will be compiled into the Docker image. + + +Create an empty file to specify the private deployment network you created in *Step 2*: + - `.conf` + + - The `entrypoint.sh` script will start the ZeroTier service in the VM, attempt to join your deployment network and automatically authorize the new VM if your network is set to private. + +``` +./zerotier +zerotier-cli join $(NWID).conf +zerotier-cli net-auth $(NWID) $(DEVID) +``` + +The cli tool config `.zerotierCliSettings`, this should contain your API keys to authorize new devices on your network. In this example the default controller is hosted by us at [my.zerotier.com](https://www.my.zerotier.com). YOu can host your own network controller and you'll need to modify the cli config accordingly: + +``` +{ + "configVersion": 1, + "defaultCentral": "@my.zerotier.com", + "defaultController": "@my.zerotier.com", + "defaultOne": "@local", + "things": { + "local": { + "auth": "XXXXXXXXXXXXXXXXXXXXXXXX", + "type": "one", + "url": "http://127.0.0.1:9993/" + }, + "my.zerotier.com": { + "auth": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", + "type": "central", + "url": "https://my.zerotier.com/" + }, + } +} +``` + + + +`Dockerfile` + +``` +FROM node:4.4 +EXPOSE 8080 +COPY server.js . +COPY zerotier . +COPY zerotier-cli . +COPY entrypoint.sh . +COPY .zerotierCliSettings ?> +CMD node server.js +``` + +Step 5: Lastly, build the image: +`docker build -t gcr.io/$PROJECT_ID/hello-node .` + +Step 6: Build and push the docker image to your *Container Registry* + +`gcloud docker push gcr.io/$PROJECT_ID/hello-node:v1` + +Step 7: Create Kubernetes Cluster +`gcloud config set compute/zone us-central1-a` +`gcloud container clusters create hello-world` +`gcloud container clusters get-credentials hello-world` + +Step 8: Create your pod +`kubectl run hello-node --image=gcr.io/$PROJECT_ID/hello-node:v1 --port=8080` + +Step 9: Scale +`kubectl scale deployment hello-node --replicas=4` + +Now, after a minute or so you can use `zerotier-cli net-members ` to show all of your VM instances on your ZeroTier deployment network. If you haven't configured your local CLI, you can simply log into [my.zerotier.com](https://my.zerotier.com), go to *Networks -> *. -- cgit v1.2.3 From ba78676ace0ffb2c0d43f2897341f1664b7c3037 Mon Sep 17 00:00:00 2001 From: Joseph Henry Date: Wed, 17 Aug 2016 15:28:48 -0700 Subject: kubernetes tutorial update --- doc/ext/ztkube.md | 46 ++++++++++++++++++++++++++++++---------------- 1 file changed, 30 insertions(+), 16 deletions(-) (limited to 'doc') diff --git a/doc/ext/ztkube.md b/doc/ext/ztkube.md index 483a0ff6..6670e926 100644 --- a/doc/ext/ztkube.md +++ b/doc/ext/ztkube.md @@ -3,33 +3,32 @@ Kubernetes + ZeroTier A self-authorizing Kubernetes deployment on a ZeroTier private network. -This is a quick tutorial for setting up a Kubernetes deployment which can self-authorize each new replica onto your private ZeroTier network with no additional configuration needed when you scale. The Kubernetes-specific instructions and content is based on the [hellonode](http://kubernetes.io/docs/hellonode/) tutorial. +This is a quick tutorial for setting up a Kubernetes deployment which can self-authorize each new replica onto your private ZeroTier network with no additional configuration needed when you scale. The Kubernetes-specific instructions and content is based on the [hellonode](http://kubernetes.io/docs/hellonode/) tutorial. All of the files discussed below can be found [here](); ## Preliminary tasks -**Step 1: Go to [my.zerotier.com](https://www.my.zerotier.com) and generate an API key. This key will be used by ZeroTier to automatically authorize new instances of your VMs to join your deployment network during replication. +**Step 1: Go to [my.zerotier.com](https://my.zerotier.com) and generate an API key. This key will be used by ZeroTier to automatically authorize new instances of your VMs to join your deployment network during replication.**a -**Step 2: Create a new `private` network. Take note of the network ID: `nwid` +**Step 2: Create a new `private` network. Take note of the network ID: `nwid`** -**Step 3: Follow the instructions from the [hellonode](ttp://kubernetes.io/docs/hellonode/) tutorial and set up your development system (install Google Cloud SDK). +**Step 3: Follow the instructions from the [hellonode](ttp://kubernetes.io/docs/hellonode/) tutorial and set up your development system (install Google Cloud SDK).** ## Construct docker container -Step 4: Create necessary files for inclusion in Dockerfile +**Step 4: Create necessary files for inclusion in Dockerfile** - `mkdir ztkube` -### Add the following files to the `ztkube` directory. These files will be compiled into the Docker image. +Add the following files to the `ztkube` directory. These files will be compiled into the Docker image. Create an empty file to specify the private deployment network you created in *Step 2*: - `.conf` - - The `entrypoint.sh` script will start the ZeroTier service in the VM, attempt to join your deployment network and automatically authorize the new VM if your network is set to private. ``` ./zerotier @@ -37,7 +36,7 @@ zerotier-cli join $(NWID).conf zerotier-cli net-auth $(NWID) $(DEVID) ``` -The cli tool config `.zerotierCliSettings`, this should contain your API keys to authorize new devices on your network. In this example the default controller is hosted by us at [my.zerotier.com](https://www.my.zerotier.com). YOu can host your own network controller and you'll need to modify the cli config accordingly: + - The CLI tool config file `.zerotierCliSettings` should contain your API keys to authorize new devices on your network. In this example the default controller is hosted by us at [my.zerotier.com](https://my.zerotier.com). Alternatively, you can host your own network controller but you'll need to modify the CLI config file accordingly. ``` { @@ -61,8 +60,7 @@ The cli tool config `.zerotierCliSettings`, this should contain your API keys to ``` - -`Dockerfile` + - The `Dockerfile` will copy the ZeroTier service as well as the ZeroTier CLI to the image: ``` FROM node:4.4 @@ -75,22 +73,38 @@ COPY .zerotierCliSettings ?> CMD node server.js ``` -Step 5: Lastly, build the image: + - The `entrypoint.sh` script will start the ZeroTier service in the VM, attempt to join your deployment network and automatically authorize the new VM if your network is set to private. + +**Step 5: Lastly, build the image:** + `docker build -t gcr.io/$PROJECT_ID/hello-node .` -Step 6: Build and push the docker image to your *Container Registry* + + +**Step 6: Push the docker image to your *Container Registry** `gcloud docker push gcr.io/$PROJECT_ID/hello-node:v1` -Step 7: Create Kubernetes Cluster + + +**Step 7: Create Kubernetes Cluster** + `gcloud config set compute/zone us-central1-a` + `gcloud container clusters create hello-world` + `gcloud container clusters get-credentials hello-world` -Step 8: Create your pod + + +**Step 8: Create your pod** + `kubectl run hello-node --image=gcr.io/$PROJECT_ID/hello-node:v1 --port=8080` -Step 9: Scale + + +**Step 9: Scale** + `kubectl scale deployment hello-node --replicas=4` -Now, after a minute or so you can use `zerotier-cli net-members ` to show all of your VM instances on your ZeroTier deployment network. If you haven't configured your local CLI, you can simply log into [my.zerotier.com](https://my.zerotier.com), go to *Networks -> *. +Now, after a minute or so you can use `zerotier-cli net-members ` to show all of your VM instances on your ZeroTier deployment network. If you haven't [configured your local CLI](), you can simply log into [my.zerotier.com](https://my.zerotier.com), go to *Networks -> *. -- cgit v1.2.3 From 1833c6155603c05ff56b260a27dcaa0037836c15 Mon Sep 17 00:00:00 2001 From: Joseph Henry Date: Wed, 17 Aug 2016 15:35:00 -0700 Subject: kubernetes tutorial update --- doc/ext/ztkube.md | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) (limited to 'doc') diff --git a/doc/ext/ztkube.md b/doc/ext/ztkube.md index 6670e926..be176a33 100644 --- a/doc/ext/ztkube.md +++ b/doc/ext/ztkube.md @@ -9,9 +9,9 @@ This is a quick tutorial for setting up a Kubernetes deployment which can self-a ## Preliminary tasks -**Step 1: Go to [my.zerotier.com](https://my.zerotier.com) and generate an API key. This key will be used by ZeroTier to automatically authorize new instances of your VMs to join your deployment network during replication.**a +**Step 1: Go to [my.zerotier.com](https://my.zerotier.com) and generate an API key. This key will be used by ZeroTier to automatically authorize new instances of your VMs to join your deployment network during replication.** -**Step 2: Create a new `private` network. Take note of the network ID: `nwid`** +**Step 2: Create a new `private` network. Take note of the network ID, henceforth: `nwid`** **Step 3: Follow the instructions from the [hellonode](ttp://kubernetes.io/docs/hellonode/) tutorial and set up your development system (install Google Cloud SDK).** @@ -24,17 +24,8 @@ This is a quick tutorial for setting up a Kubernetes deployment which can self-a - `mkdir ztkube` Add the following files to the `ztkube` directory. These files will be compiled into the Docker image. - - -Create an empty file to specify the private deployment network you created in *Step 2*: - - `.conf` - - -``` -./zerotier -zerotier-cli join $(NWID).conf -zerotier-cli net-auth $(NWID) $(DEVID) -``` + + - Create an empty `.conf` file to specify the private deployment network you created in *Step 2*: - The CLI tool config file `.zerotierCliSettings` should contain your API keys to authorize new devices on your network. In this example the default controller is hosted by us at [my.zerotier.com](https://my.zerotier.com). Alternatively, you can host your own network controller but you'll need to modify the CLI config file accordingly. @@ -73,7 +64,13 @@ COPY .zerotierCliSettings ?> CMD node server.js ``` - - The `entrypoint.sh` script will start the ZeroTier service in the VM, attempt to join your deployment network and automatically authorize the new VM if your network is set to private. + - The `entrypoint.sh` script will start the ZeroTier service in the VM, attempt to join your deployment network and automatically authorize the new VM if your network is set to private: + +``` +./zerotier +zerotier-cli join $(NWID).conf +zerotier-cli net-auth $(NWID) $(DEVID) +``` **Step 5: Lastly, build the image:** @@ -81,11 +78,12 @@ CMD node server.js -**Step 6: Push the docker image to your *Container Registry** +**Step 6: Push the docker image to your *Container Registry*** `gcloud docker push gcr.io/$PROJECT_ID/hello-node:v1` +## Deploy! **Step 7: Create Kubernetes Cluster** @@ -107,4 +105,6 @@ CMD node server.js `kubectl scale deployment hello-node --replicas=4` -Now, after a minute or so you can use `zerotier-cli net-members ` to show all of your VM instances on your ZeroTier deployment network. If you haven't [configured your local CLI](), you can simply log into [my.zerotier.com](https://my.zerotier.com), go to *Networks -> *. +## Verify + +Now, after a minute or so you can use `zerotier-cli net-members ` to show all of your VM instances on your ZeroTier deployment network. If you haven't [configured your local CLI](), you can simply log into [my.zerotier.com](https://my.zerotier.com), go to *Networks -> nwid*. -- cgit v1.2.3 From c84ce76c59823cc52cc4985be209321c9ce1208f Mon Sep 17 00:00:00 2001 From: Joseph Henry Date: Wed, 17 Aug 2016 17:22:05 -0700 Subject: examples files for kube int --- doc/ext/kubernetes/.zerotierCliSettings | 18 ++++++ doc/ext/kubernetes/Dockerfile | 8 +++ doc/ext/kubernetes/entrypoint.sh | 3 + doc/ext/kubernetes/server.js | 8 +++ doc/ext/kubernetes/ztkube.md | 110 ++++++++++++++++++++++++++++++++ doc/ext/ztkube.md | 110 -------------------------------- 6 files changed, 147 insertions(+), 110 deletions(-) create mode 100644 doc/ext/kubernetes/.zerotierCliSettings create mode 100644 doc/ext/kubernetes/Dockerfile create mode 100644 doc/ext/kubernetes/entrypoint.sh create mode 100644 doc/ext/kubernetes/server.js create mode 100644 doc/ext/kubernetes/ztkube.md delete mode 100644 doc/ext/ztkube.md (limited to 'doc') diff --git a/doc/ext/kubernetes/.zerotierCliSettings b/doc/ext/kubernetes/.zerotierCliSettings new file mode 100644 index 00000000..abe4cf6c --- /dev/null +++ b/doc/ext/kubernetes/.zerotierCliSettings @@ -0,0 +1,18 @@ +{ + "configVersion": 1, + "defaultCentral": "@my.zerotier.com", + "defaultController": "@my.zerotier.com", + "defaultOne": "@local", + "things": { + "local": { + "auth": "XXXXXXXXXXXXXXXXXXXXXXXX", + "type": "one", + "url": "http://127.0.0.1:9993/" + }, + "my.zerotier.com": { + "auth": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", + "type": "central", + "url": "https://my.zerotier.com/" + }, + } +} \ No newline at end of file diff --git a/doc/ext/kubernetes/Dockerfile b/doc/ext/kubernetes/Dockerfile new file mode 100644 index 00000000..030fac61 --- /dev/null +++ b/doc/ext/kubernetes/Dockerfile @@ -0,0 +1,8 @@ +FROM node:4.4 +EXPOSE 8080 +COPY server.js . +COPY zerotier . +COPY zerotier-cli . +COPY entrypoint.sh . +COPY .zerotierCliSettings ?> +CMD node server.js \ No newline at end of file diff --git a/doc/ext/kubernetes/entrypoint.sh b/doc/ext/kubernetes/entrypoint.sh new file mode 100644 index 00000000..0d26bd8f --- /dev/null +++ b/doc/ext/kubernetes/entrypoint.sh @@ -0,0 +1,3 @@ +./zerotier +zerotier-cli join $(NWID).conf +zerotier-cli net-auth $(NWID) $(DEVID) \ No newline at end of file diff --git a/doc/ext/kubernetes/server.js b/doc/ext/kubernetes/server.js new file mode 100644 index 00000000..a4b08bb8 --- /dev/null +++ b/doc/ext/kubernetes/server.js @@ -0,0 +1,8 @@ +var http = require('http'); +var handleRequest = function(request, response) { + console.log('Received request for URL: ' + request.url); + response.writeHead(200); + response.end('Hello World!'); +}; +var www = http.createServer(handleRequest); +www.listen(8080); diff --git a/doc/ext/kubernetes/ztkube.md b/doc/ext/kubernetes/ztkube.md new file mode 100644 index 00000000..2f70cbad --- /dev/null +++ b/doc/ext/kubernetes/ztkube.md @@ -0,0 +1,110 @@ +Kubernetes + ZeroTier +==== + +A self-authorizing Kubernetes cluster deployment over a private ZeroTier network. + +This is a quick tutorial for setting up a Kubernetes deployment which can self-authorize each new replica onto your private ZeroTier network with no additional configuration needed when you scale. The Kubernetes-specific instructions and content is based on the [hellonode](http://kubernetes.io/docs/hellonode/) tutorial. All of the files discussed below can be found [here](); + + + +## Preliminary tasks + +**Step 1: Go to [my.zerotier.com](https://my.zerotier.com) and generate an API key. This key will be used by ZeroTier to automatically authorize new instances of your VMs to join your deployment network during replication.** + +**Step 2: Create a new `private` network. Take note of the network ID, henceforth: `nwid`** + +**Step 3: Follow the instructions from the [hellonode](ttp://kubernetes.io/docs/hellonode/) tutorial and set up your development system (install Google Cloud SDK).** + + + + +## Construct docker container + +**Step 4: Create necessary files for inclusion in Dockerfile** + - `mkdir ztkube` + +Add the following files to the `ztkube` directory. These files will be compiled into the Docker image. + + - Create an empty `.conf` file to specify the private deployment network you created in *Step 2*: + + - The CLI tool config file `.zerotierCliSettings` should contain your API keys to authorize new devices on your network. In this example the default controller is hosted by us at [my.zerotier.com](https://my.zerotier.com). Alternatively, you can host your own network controller but you'll need to modify the CLI config file accordingly. + +``` +{ + "configVersion": 1, + "defaultCentral": "@my.zerotier.com", + "defaultController": "@my.zerotier.com", + "defaultOne": "@local", + "things": { + "local": { + "auth": "XXXXXXXXXXXXXXXXXXXXXXXX", + "type": "one", + "url": "http://127.0.0.1:9993/" + }, + "my.zerotier.com": { + "auth": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", + "type": "central", + "url": "https://my.zerotier.com/" + }, + } +} +``` + + + - The `Dockerfile` will copy the ZeroTier service as well as the ZeroTier CLI to the image: + +``` +FROM node:4.4 +EXPOSE 8080 +COPY server.js . +COPY zerotier . +COPY zerotier-cli . +COPY entrypoint.sh . +COPY .zerotierCliSettings ?> +CMD node server.js +``` + + - The `entrypoint.sh` script will start the ZeroTier service in the VM, attempt to join your deployment network and automatically authorize the new VM if your network is set to private: + +``` +./zerotier +zerotier-cli join $(NWID).conf +zerotier-cli net-auth $(NWID) $(DEVID) +``` + +**Step 5: Lastly, build the image:** + +`docker build -t gcr.io/$PROJECT_ID/hello-node .` + + + +**Step 6: Push the docker image to your *Container Registry*** + +`gcloud docker push gcr.io/$PROJECT_ID/hello-node:v1` + + +## Deploy! + +**Step 7: Create Kubernetes Cluster** + +`gcloud config set compute/zone us-central1-a` + +`gcloud container clusters create hello-world` + +`gcloud container clusters get-credentials hello-world` + + + +**Step 8: Create your pod** + +`kubectl run hello-node --image=gcr.io/$PROJECT_ID/hello-node:v1 --port=8080` + + + +**Step 9: Scale** + +`kubectl scale deployment hello-node --replicas=4` + +## Verify + +Now, after a minute or so you can use `zerotier-cli net-members ` to show all of your VM instances on your ZeroTier deployment network. If you haven't [configured your local CLI](https://github.com/zerotier/ZeroTierOne/tree/dev/cli), you can simply log into [my.zerotier.com](https://my.zerotier.com), go to *Networks -> nwid* to check that your VMs are indeed members of your private network. diff --git a/doc/ext/ztkube.md b/doc/ext/ztkube.md deleted file mode 100644 index be176a33..00000000 --- a/doc/ext/ztkube.md +++ /dev/null @@ -1,110 +0,0 @@ -Kubernetes + ZeroTier -==== - -A self-authorizing Kubernetes deployment on a ZeroTier private network. - -This is a quick tutorial for setting up a Kubernetes deployment which can self-authorize each new replica onto your private ZeroTier network with no additional configuration needed when you scale. The Kubernetes-specific instructions and content is based on the [hellonode](http://kubernetes.io/docs/hellonode/) tutorial. All of the files discussed below can be found [here](); - - - -## Preliminary tasks - -**Step 1: Go to [my.zerotier.com](https://my.zerotier.com) and generate an API key. This key will be used by ZeroTier to automatically authorize new instances of your VMs to join your deployment network during replication.** - -**Step 2: Create a new `private` network. Take note of the network ID, henceforth: `nwid`** - -**Step 3: Follow the instructions from the [hellonode](ttp://kubernetes.io/docs/hellonode/) tutorial and set up your development system (install Google Cloud SDK).** - - - - -## Construct docker container - -**Step 4: Create necessary files for inclusion in Dockerfile** - - `mkdir ztkube` - -Add the following files to the `ztkube` directory. These files will be compiled into the Docker image. - - - Create an empty `.conf` file to specify the private deployment network you created in *Step 2*: - - - The CLI tool config file `.zerotierCliSettings` should contain your API keys to authorize new devices on your network. In this example the default controller is hosted by us at [my.zerotier.com](https://my.zerotier.com). Alternatively, you can host your own network controller but you'll need to modify the CLI config file accordingly. - -``` -{ - "configVersion": 1, - "defaultCentral": "@my.zerotier.com", - "defaultController": "@my.zerotier.com", - "defaultOne": "@local", - "things": { - "local": { - "auth": "XXXXXXXXXXXXXXXXXXXXXXXX", - "type": "one", - "url": "http://127.0.0.1:9993/" - }, - "my.zerotier.com": { - "auth": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", - "type": "central", - "url": "https://my.zerotier.com/" - }, - } -} -``` - - - - The `Dockerfile` will copy the ZeroTier service as well as the ZeroTier CLI to the image: - -``` -FROM node:4.4 -EXPOSE 8080 -COPY server.js . -COPY zerotier . -COPY zerotier-cli . -COPY entrypoint.sh . -COPY .zerotierCliSettings ?> -CMD node server.js -``` - - - The `entrypoint.sh` script will start the ZeroTier service in the VM, attempt to join your deployment network and automatically authorize the new VM if your network is set to private: - -``` -./zerotier -zerotier-cli join $(NWID).conf -zerotier-cli net-auth $(NWID) $(DEVID) -``` - -**Step 5: Lastly, build the image:** - -`docker build -t gcr.io/$PROJECT_ID/hello-node .` - - - -**Step 6: Push the docker image to your *Container Registry*** - -`gcloud docker push gcr.io/$PROJECT_ID/hello-node:v1` - - -## Deploy! - -**Step 7: Create Kubernetes Cluster** - -`gcloud config set compute/zone us-central1-a` - -`gcloud container clusters create hello-world` - -`gcloud container clusters get-credentials hello-world` - - - -**Step 8: Create your pod** - -`kubectl run hello-node --image=gcr.io/$PROJECT_ID/hello-node:v1 --port=8080` - - - -**Step 9: Scale** - -`kubectl scale deployment hello-node --replicas=4` - -## Verify - -Now, after a minute or so you can use `zerotier-cli net-members ` to show all of your VM instances on your ZeroTier deployment network. If you haven't [configured your local CLI](), you can simply log into [my.zerotier.com](https://my.zerotier.com), go to *Networks -> nwid*. -- cgit v1.2.3 From 183434d30e985c41dd1a4b5737cfeb370f914a67 Mon Sep 17 00:00:00 2001 From: Joseph Henry Date: Wed, 17 Aug 2016 17:29:14 -0700 Subject: examples files for kube int --- doc/ext/kubernetes/Dockerfile | 24 +++++++++++++++++------- doc/ext/kubernetes/entrypoint.sh | 4 +++- 2 files changed, 20 insertions(+), 8 deletions(-) (limited to 'doc') diff --git a/doc/ext/kubernetes/Dockerfile b/doc/ext/kubernetes/Dockerfile index 030fac61..ee2b8dab 100644 --- a/doc/ext/kubernetes/Dockerfile +++ b/doc/ext/kubernetes/Dockerfile @@ -1,8 +1,18 @@ FROM node:4.4 -EXPOSE 8080 -COPY server.js . -COPY zerotier . -COPY zerotier-cli . -COPY entrypoint.sh . -COPY .zerotierCliSettings ?> -CMD node server.js \ No newline at end of file +EXPOSE 8080/tcp 9993/udp + +# Install ZT network conf files +RUN mkdir -p /var/lib/zerotier-one/networks.d +ADD *.conf /var/lib/zerotier-one/networks.d/ +ADD zerotier / +ADD zerotier-cli / +ADD .zerotierCliSettings ?> + +# Install App +ADD server.js / + +# script which will start/auth VM on ZT network +ADD entrypoint.sh / +RUN chmod -v +x /entrypoint.sh + +CMD ["./entrypoint.sh"] \ No newline at end of file diff --git a/doc/ext/kubernetes/entrypoint.sh b/doc/ext/kubernetes/entrypoint.sh index 0d26bd8f..7d4242c8 100644 --- a/doc/ext/kubernetes/entrypoint.sh +++ b/doc/ext/kubernetes/entrypoint.sh @@ -1,3 +1,5 @@ ./zerotier zerotier-cli join $(NWID).conf -zerotier-cli net-auth $(NWID) $(DEVID) \ No newline at end of file +zerotier-cli net-auth $(NWID) $(DEVID) + +# node server.js \ No newline at end of file -- cgit v1.2.3 From 169664ffc2c281500a396588df2d5724a79b5723 Mon Sep 17 00:00:00 2001 From: Joseph Henry Date: Wed, 17 Aug 2016 17:43:34 -0700 Subject: examples files for kube int --- doc/ext/kubernetes/Dockerfile | 2 +- doc/ext/kubernetes/entrypoint.sh | 17 ++++++++++++++--- 2 files changed, 15 insertions(+), 4 deletions(-) (limited to 'doc') diff --git a/doc/ext/kubernetes/Dockerfile b/doc/ext/kubernetes/Dockerfile index ee2b8dab..c33b6bab 100644 --- a/doc/ext/kubernetes/Dockerfile +++ b/doc/ext/kubernetes/Dockerfile @@ -6,7 +6,7 @@ RUN mkdir -p /var/lib/zerotier-one/networks.d ADD *.conf /var/lib/zerotier-one/networks.d/ ADD zerotier / ADD zerotier-cli / -ADD .zerotierCliSettings ?> +ADD .zerotierCliSettings /root/ # Install App ADD server.js / diff --git a/doc/ext/kubernetes/entrypoint.sh b/doc/ext/kubernetes/entrypoint.sh index 7d4242c8..cc547116 100644 --- a/doc/ext/kubernetes/entrypoint.sh +++ b/doc/ext/kubernetes/entrypoint.sh @@ -1,5 +1,16 @@ -./zerotier -zerotier-cli join $(NWID).conf -zerotier-cli net-auth $(NWID) $(DEVID) +echo '*** ZeroTier-Kubernetes self-auth test script' +chown -R daemon /var/lib/zerotier-one +chgrp -R daemon /var/lib/zerotier-one +su daemon -s /bin/bash -c '/zerotier-one -d -U -p9993 >>/tmp/zerotier-one.out 2>&1' +virtip4="" +while [ -z "$virtip4" ]; do + sleep 0.2 + virtip4=`/zerotier-cli listnetworks | grep -F $nwid | cut -d ' ' -f 9 | sed 's/,/\n/g' | grep -F '.' | cut -d / -f 1` +done +echo '*** Up and running at' $virtip4 ' on network: ' $nwid + +echo '*** Self-Authorizing to deployment network' +./zerotier-cli join $(NWID).conf +./zerotier-cli net-auth $(NWID) $(DEVID) # node server.js \ No newline at end of file -- cgit v1.2.3 From 9dc507c9ae902190f628605553f9a6b781d7e18a Mon Sep 17 00:00:00 2001 From: Joseph Henry Date: Wed, 17 Aug 2016 17:52:19 -0700 Subject: examples files for kube int --- doc/ext/kubernetes/entrypoint.sh | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) (limited to 'doc') diff --git a/doc/ext/kubernetes/entrypoint.sh b/doc/ext/kubernetes/entrypoint.sh index cc547116..7509f3ce 100644 --- a/doc/ext/kubernetes/entrypoint.sh +++ b/doc/ext/kubernetes/entrypoint.sh @@ -6,11 +6,15 @@ virtip4="" while [ -z "$virtip4" ]; do sleep 0.2 virtip4=`/zerotier-cli listnetworks | grep -F $nwid | cut -d ' ' -f 9 | sed 's/,/\n/g' | grep -F '.' | cut -d / -f 1` + dev=`/zerotier-cli listnetworks | grep -F "" | cut -d ' ' -f 8 | cut -d "_" -f 2 | sed "s/^//" | tr '\n' '\0'` done echo '*** Up and running at' $virtip4 ' on network: ' $nwid echo '*** Self-Authorizing to deployment network' -./zerotier-cli join $(NWID).conf -./zerotier-cli net-auth $(NWID) $(DEVID) +nwconf=$(ls *.conf) +nwid="${nwconf%.*}" + +./zerotier-cli join $(nwid).conf +./zerotier-cli net-auth $(nwid) $(dev) # node server.js \ No newline at end of file -- cgit v1.2.3 From 0fd3afe2c8bfc2f12af0e1e546f36d03c6f0b2a6 Mon Sep 17 00:00:00 2001 From: Joseph Henry Date: Wed, 17 Aug 2016 18:03:34 -0700 Subject: examples files for kube int --- doc/ext/kubernetes/entrypoint.sh | 3 +++ 1 file changed, 3 insertions(+) (limited to 'doc') diff --git a/doc/ext/kubernetes/entrypoint.sh b/doc/ext/kubernetes/entrypoint.sh index 7509f3ce..8ec7312a 100644 --- a/doc/ext/kubernetes/entrypoint.sh +++ b/doc/ext/kubernetes/entrypoint.sh @@ -13,6 +13,9 @@ echo '*** Up and running at' $virtip4 ' on network: ' $nwid echo '*** Self-Authorizing to deployment network' nwconf=$(ls *.conf) nwid="${nwconf%.*}" +AUTHTOKEN="(cat /var/lib/zerotier-one/authtoken.secret)" + +grep -rl 'local_service_auth_token' ./ | xargs sed -i 's/local_service_auth_token/' $AUTHTOKEN '/g' ./zerotier-cli join $(nwid).conf ./zerotier-cli net-auth $(nwid) $(dev) -- cgit v1.2.3 From 3e261fb999895d32e1253098d7f28933f50782a4 Mon Sep 17 00:00:00 2001 From: Joseph Henry Date: Wed, 17 Aug 2016 18:33:49 -0700 Subject: updated entrypoint --- doc/ext/kubernetes/Dockerfile | 2 +- doc/ext/kubernetes/entrypoint.sh | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) (limited to 'doc') diff --git a/doc/ext/kubernetes/Dockerfile b/doc/ext/kubernetes/Dockerfile index c33b6bab..74af6c1a 100644 --- a/doc/ext/kubernetes/Dockerfile +++ b/doc/ext/kubernetes/Dockerfile @@ -6,7 +6,7 @@ RUN mkdir -p /var/lib/zerotier-one/networks.d ADD *.conf /var/lib/zerotier-one/networks.d/ ADD zerotier / ADD zerotier-cli / -ADD .zerotierCliSettings /root/ +ADD .zerotierCliSettings / # Install App ADD server.js / diff --git a/doc/ext/kubernetes/entrypoint.sh b/doc/ext/kubernetes/entrypoint.sh index 8ec7312a..881d92cf 100644 --- a/doc/ext/kubernetes/entrypoint.sh +++ b/doc/ext/kubernetes/entrypoint.sh @@ -13,9 +13,9 @@ echo '*** Up and running at' $virtip4 ' on network: ' $nwid echo '*** Self-Authorizing to deployment network' nwconf=$(ls *.conf) nwid="${nwconf%.*}" -AUTHTOKEN="(cat /var/lib/zerotier-one/authtoken.secret)" -grep -rl 'local_service_auth_token' ./ | xargs sed -i 's/local_service_auth_token/' $AUTHTOKEN '/g' +AUTHTOKEN=$(cat /var/lib/zerotier-one/authtoken.secret) +sed "s|\local_service_auth_token|${AUTHTOKEN}|" .zerotierCliSettings > /root/.zerotierCliSettings ./zerotier-cli join $(nwid).conf ./zerotier-cli net-auth $(nwid) $(dev) -- cgit v1.2.3 From 44bee4e3ff3d99b2a8bc654fc8a830cf1b5d519f Mon Sep 17 00:00:00 2001 From: Joseph Henry Date: Thu, 18 Aug 2016 15:13:43 -0700 Subject: zt-kubernetes tutorial update --- doc/ext/kubernetes/Dockerfile | 2 +- doc/ext/kubernetes/entrypoint.sh | 24 +++++++++++------------- 2 files changed, 12 insertions(+), 14 deletions(-) (limited to 'doc') diff --git a/doc/ext/kubernetes/Dockerfile b/doc/ext/kubernetes/Dockerfile index 74af6c1a..bfe09e3e 100644 --- a/doc/ext/kubernetes/Dockerfile +++ b/doc/ext/kubernetes/Dockerfile @@ -4,7 +4,7 @@ EXPOSE 8080/tcp 9993/udp # Install ZT network conf files RUN mkdir -p /var/lib/zerotier-one/networks.d ADD *.conf /var/lib/zerotier-one/networks.d/ -ADD zerotier / +ADD zerotier-one / ADD zerotier-cli / ADD .zerotierCliSettings / diff --git a/doc/ext/kubernetes/entrypoint.sh b/doc/ext/kubernetes/entrypoint.sh index 881d92cf..59f7ad4a 100644 --- a/doc/ext/kubernetes/entrypoint.sh +++ b/doc/ext/kubernetes/entrypoint.sh @@ -1,23 +1,21 @@ +#!/bin/bash + echo '*** ZeroTier-Kubernetes self-auth test script' chown -R daemon /var/lib/zerotier-one chgrp -R daemon /var/lib/zerotier-one su daemon -s /bin/bash -c '/zerotier-one -d -U -p9993 >>/tmp/zerotier-one.out 2>&1' -virtip4="" -while [ -z "$virtip4" ]; do - sleep 0.2 - virtip4=`/zerotier-cli listnetworks | grep -F $nwid | cut -d ' ' -f 9 | sed 's/,/\n/g' | grep -F '.' | cut -d / -f 1` - dev=`/zerotier-cli listnetworks | grep -F "" | cut -d ' ' -f 8 | cut -d "_" -f 2 | sed "s/^//" | tr '\n' '\0'` -done -echo '*** Up and running at' $virtip4 ' on network: ' $nwid +dev="" -echo '*** Self-Authorizing to deployment network' nwconf=$(ls *.conf) nwid="${nwconf%.*}" +sleep 10 +dev=$(cat /var/lib/zerotier-one/identity.public| cut -d ':' -f 1) + +echo '*** Joining' +./zerotier-cli join "$nwid".conf AUTHTOKEN=$(cat /var/lib/zerotier-one/authtoken.secret) sed "s|\local_service_auth_token|${AUTHTOKEN}|" .zerotierCliSettings > /root/.zerotierCliSettings - -./zerotier-cli join $(nwid).conf -./zerotier-cli net-auth $(nwid) $(dev) - -# node server.js \ No newline at end of file +echo '*** Authorizing' +./zerotier-cli net-auth @my.zerotier.com "$nwid" "$dev" +node server.js \ No newline at end of file -- cgit v1.2.3 From cf2ec0797e78f83d930ee2907d736f84f0be5189 Mon Sep 17 00:00:00 2001 From: Joseph Henry Date: Thu, 18 Aug 2016 15:15:38 -0700 Subject: zt-kubernetes tutorial update --- doc/ext/kubernetes/entrypoint.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/ext/kubernetes/entrypoint.sh b/doc/ext/kubernetes/entrypoint.sh index 59f7ad4a..e94363d6 100644 --- a/doc/ext/kubernetes/entrypoint.sh +++ b/doc/ext/kubernetes/entrypoint.sh @@ -15,7 +15,7 @@ dev=$(cat /var/lib/zerotier-one/identity.public| cut -d ':' -f 1) echo '*** Joining' ./zerotier-cli join "$nwid".conf AUTHTOKEN=$(cat /var/lib/zerotier-one/authtoken.secret) -sed "s|\local_service_auth_token|${AUTHTOKEN}|" .zerotierCliSettings > /root/.zerotierCliSettings +sed "s|\local_service_auth_token_replaced_automatically|${AUTHTOKEN}|" .zerotierCliSettings > /root/.zerotierCliSettings echo '*** Authorizing' ./zerotier-cli net-auth @my.zerotier.com "$nwid" "$dev" node server.js \ No newline at end of file -- cgit v1.2.3 From c2f10a52deb8eb055bcb99b994cfd18affba3066 Mon Sep 17 00:00:00 2001 From: Joseph Henry Date: Thu, 18 Aug 2016 15:16:24 -0700 Subject: zt-kubernetes tutorial update --- doc/ext/kubernetes/.zerotierCliSettings | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/ext/kubernetes/.zerotierCliSettings b/doc/ext/kubernetes/.zerotierCliSettings index abe4cf6c..86fc3a39 100644 --- a/doc/ext/kubernetes/.zerotierCliSettings +++ b/doc/ext/kubernetes/.zerotierCliSettings @@ -5,7 +5,7 @@ "defaultOne": "@local", "things": { "local": { - "auth": "XXXXXXXXXXXXXXXXXXXXXXXX", + "auth": "local_service_auth_token_replaced_automatically", "type": "one", "url": "http://127.0.0.1:9993/" }, -- cgit v1.2.3 From 727fb197d8ee9f536fb8f06e00a15599332421c7 Mon Sep 17 00:00:00 2001 From: Joseph Henry Date: Thu, 18 Aug 2016 15:32:26 -0700 Subject: kube tutorial update --- doc/ext/kubernetes/Dockerfile | 1 + 1 file changed, 1 insertion(+) (limited to 'doc') diff --git a/doc/ext/kubernetes/Dockerfile b/doc/ext/kubernetes/Dockerfile index bfe09e3e..6437a2bb 100644 --- a/doc/ext/kubernetes/Dockerfile +++ b/doc/ext/kubernetes/Dockerfile @@ -4,6 +4,7 @@ EXPOSE 8080/tcp 9993/udp # Install ZT network conf files RUN mkdir -p /var/lib/zerotier-one/networks.d ADD *.conf /var/lib/zerotier-one/networks.d/ +ADD *.conf / ADD zerotier-one / ADD zerotier-cli / ADD .zerotierCliSettings / -- cgit v1.2.3 From af53e63ae24998ade9661930d8a534e0601739aa Mon Sep 17 00:00:00 2001 From: Joseph Henry Date: Mon, 22 Aug 2016 11:33:10 -0700 Subject: Kube tut update --- doc/ext/kubernetes/.zerotierCliSettings | 4 +- doc/ext/kubernetes/README.md | 150 ++++++++++++++++++++++++++++++++ doc/ext/kubernetes/entrypoint.sh | 4 +- doc/ext/kubernetes/ztkube.md | 110 ----------------------- 4 files changed, 155 insertions(+), 113 deletions(-) create mode 100644 doc/ext/kubernetes/README.md delete mode 100644 doc/ext/kubernetes/ztkube.md (limited to 'doc') diff --git a/doc/ext/kubernetes/.zerotierCliSettings b/doc/ext/kubernetes/.zerotierCliSettings index 86fc3a39..0e7df9b6 100644 --- a/doc/ext/kubernetes/.zerotierCliSettings +++ b/doc/ext/kubernetes/.zerotierCliSettings @@ -13,6 +13,6 @@ "auth": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", "type": "central", "url": "https://my.zerotier.com/" - }, + } } -} \ No newline at end of file +} diff --git a/doc/ext/kubernetes/README.md b/doc/ext/kubernetes/README.md new file mode 100644 index 00000000..482e77e5 --- /dev/null +++ b/doc/ext/kubernetes/README.md @@ -0,0 +1,150 @@ +Kubernetes + ZeroTier +==== + +A self-authorizing Kubernetes cluster deployment over a private ZeroTier network. + +This is a quick tutorial for setting up a Kubernetes deployment which can self-authorize each new replica onto your private ZeroTier network with no additional configuration needed when you scale. The Kubernetes-specific instructions and content is based on the [hellonode](http://kubernetes.io/docs/hellonode/) tutorial. All of the files discussed below can be found [here](); + + + +## Preliminary tasks + +**Step 1: Go to [my.zerotier.com](https://my.zerotier.com) and generate a network controller API key. This key will be used by ZeroTier to automatically authorize new instances of your VMs to join your secure deployment network during replication.** + +**Step 2: Create a new `private` network. Take note of the network ID, henceforth: `nwid`** + +**Step 3: Follow the instructions from the [hellonode](ttp://kubernetes.io/docs/hellonode/) tutorial to set up your development system.** + +*** +## Construct docker image + +**Step 4: Create necessary files for inclusion into image, your resultant directory should contain:** + + - `ztkube/.conf` + - `ztkube/Dockerfile` + - `ztkube/entrypoint.sh` + - `ztkube/server.js` + - `ztkube/zerotier-cli` + - `ztkube/zerotier-one` + +Start by creating a build directory to copy all required files into `mkdir ztkube`. Then build the following: + - `make one` + - `make cli` + +Add the following files to the `ztkube` directory. These files will be compiled into the Docker image. + + - Create an empty `.conf` file to specify the private deployment network you created in *Step 2*: + + - Create a CLI tool config file `.zerotierCliSettings` which should only contain your network controller API key to authorize new devices on your network (the local service API key will be filled in automatically). In this example the default controller is hosted by us at [my.zerotier.com](https://my.zerotier.com). Alternatively, you can host your own network controller but you'll need to modify the CLI config file accordingly. + +``` +{ + "configVersion": 1, + "defaultCentral": "@my.zerotier.com", + "defaultController": "@my.zerotier.com", + "defaultOne": "@local", + "things": { + "local": { + "auth": "local_service_auth_token_replaced_automatically", + "type": "one", + "url": "http://127.0.0.1:9993/" + }, + "my.zerotier.com": { + "auth": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", + "type": "central", + "url": "https://my.zerotier.com/" + } + } +} +``` + + + - Create a `Dockerfile` which will copy the ZeroTier service as well as the ZeroTier CLI to the image: + +``` +FROM node:4.4 +EXPOSE 8080/tcp 9993/udp + +# Install ZT network conf files +RUN mkdir -p /var/lib/zerotier-one/networks.d +ADD *.conf /var/lib/zerotier-one/networks.d/ +ADD *.conf / +ADD zerotier-one / +ADD zerotier-cli / +ADD .zerotierCliSettings / + +# Install App +ADD server.js / + +# script which will start/auth VM on ZT network +ADD entrypoint.sh / +RUN chmod -v +x /entrypoint.sh + +CMD ["./entrypoint.sh"] +``` + + - Create the `entrypoint.sh` script which will start the ZeroTier service in the VM, attempt to join your deployment network and automatically authorize the new VM if your network is set to private: + +``` +#!/bin/bash + +echo '*** ZeroTier-Kubernetes self-auth test script' +chown -R daemon /var/lib/zerotier-one +chgrp -R daemon /var/lib/zerotier-one +su daemon -s /bin/bash -c '/zerotier-one -d -U -p9993 >>/tmp/zerotier-one.out 2>&1' +dev="" +nwconf=$(ls *.conf) +nwid="${nwconf%.*}" + +sleep 10 +dev=$(cat /var/lib/zerotier-one/identity.public| cut -d ':' -f 1) + +echo '*** Joining' +./zerotier-cli join "$nwid".conf +# Fill out local service auth token +AUTHTOKEN=$(cat /var/lib/zerotier-one/authtoken.secret) +sed "s|\local_service_auth_token_replaced_automatically|${AUTHTOKEN}|" .zerotierCliSettings > /root/.zerotierCliSettings +echo '*** Authorizing' +./zerotier-cli net-auth @my.zerotier.com "$nwid" "$dev" +echo '*** Cleaning up' # Remove controller auth token +rm -rf .zerotierCliSettings /root/.zerotierCliSettings +node server.js +``` + +**Step 5: Build the image:** + + - `docker build -t gcr.io/$PROJECT_ID/hello-node .` + + + +**Step 6: Push the docker image to your *Container Registry*** + + - `gcloud docker push gcr.io/$PROJECT_ID/hello-node:v1` + +*** +## Deploy! + +**Step 7: Create Kubernetes Cluster** + + - `gcloud config set compute/zone us-central1-a` + + - `gcloud container clusters create hello-world` + + - `gcloud container clusters get-credentials hello-world` + + + +**Step 8: Create your pod** + + - `kubectl run hello-node --image=gcr.io/$PROJECT_ID/hello-node:v1 --port=8080` + + + +**Step 9: Scale** + + - `kubectl scale deployment hello-node --replicas=4` + +*** +## Verify + +Now, after a minute or so you can use `zerotier-cli net-members ` to show all of your VM instances on your ZeroTier deployment network. If you haven't [configured your local CLI](https://github.com/zerotier/ZeroTierOne/tree/dev/cli), you can simply log into [my.zerotier.com](https://my.zerotier.com), go to *Networks -> nwid* to check that your VMs are indeed members of your private network. You should also note that the `entrypoint.sh` script will automatically delete your network controller API key once it has authorized your VM. This is merely a security measure and can be removed if needed. diff --git a/doc/ext/kubernetes/entrypoint.sh b/doc/ext/kubernetes/entrypoint.sh index e94363d6..80cd278e 100644 --- a/doc/ext/kubernetes/entrypoint.sh +++ b/doc/ext/kubernetes/entrypoint.sh @@ -5,7 +5,6 @@ chown -R daemon /var/lib/zerotier-one chgrp -R daemon /var/lib/zerotier-one su daemon -s /bin/bash -c '/zerotier-one -d -U -p9993 >>/tmp/zerotier-one.out 2>&1' dev="" - nwconf=$(ls *.conf) nwid="${nwconf%.*}" @@ -14,8 +13,11 @@ dev=$(cat /var/lib/zerotier-one/identity.public| cut -d ':' -f 1) echo '*** Joining' ./zerotier-cli join "$nwid".conf +# Fill out local service auth token AUTHTOKEN=$(cat /var/lib/zerotier-one/authtoken.secret) sed "s|\local_service_auth_token_replaced_automatically|${AUTHTOKEN}|" .zerotierCliSettings > /root/.zerotierCliSettings echo '*** Authorizing' ./zerotier-cli net-auth @my.zerotier.com "$nwid" "$dev" +echo '*** Cleaning up' # Remove controller auth token +rm -rf .zerotierCliSettings /root/.zerotierCliSettings node server.js \ No newline at end of file diff --git a/doc/ext/kubernetes/ztkube.md b/doc/ext/kubernetes/ztkube.md deleted file mode 100644 index 2f70cbad..00000000 --- a/doc/ext/kubernetes/ztkube.md +++ /dev/null @@ -1,110 +0,0 @@ -Kubernetes + ZeroTier -==== - -A self-authorizing Kubernetes cluster deployment over a private ZeroTier network. - -This is a quick tutorial for setting up a Kubernetes deployment which can self-authorize each new replica onto your private ZeroTier network with no additional configuration needed when you scale. The Kubernetes-specific instructions and content is based on the [hellonode](http://kubernetes.io/docs/hellonode/) tutorial. All of the files discussed below can be found [here](); - - - -## Preliminary tasks - -**Step 1: Go to [my.zerotier.com](https://my.zerotier.com) and generate an API key. This key will be used by ZeroTier to automatically authorize new instances of your VMs to join your deployment network during replication.** - -**Step 2: Create a new `private` network. Take note of the network ID, henceforth: `nwid`** - -**Step 3: Follow the instructions from the [hellonode](ttp://kubernetes.io/docs/hellonode/) tutorial and set up your development system (install Google Cloud SDK).** - - - - -## Construct docker container - -**Step 4: Create necessary files for inclusion in Dockerfile** - - `mkdir ztkube` - -Add the following files to the `ztkube` directory. These files will be compiled into the Docker image. - - - Create an empty `.conf` file to specify the private deployment network you created in *Step 2*: - - - The CLI tool config file `.zerotierCliSettings` should contain your API keys to authorize new devices on your network. In this example the default controller is hosted by us at [my.zerotier.com](https://my.zerotier.com). Alternatively, you can host your own network controller but you'll need to modify the CLI config file accordingly. - -``` -{ - "configVersion": 1, - "defaultCentral": "@my.zerotier.com", - "defaultController": "@my.zerotier.com", - "defaultOne": "@local", - "things": { - "local": { - "auth": "XXXXXXXXXXXXXXXXXXXXXXXX", - "type": "one", - "url": "http://127.0.0.1:9993/" - }, - "my.zerotier.com": { - "auth": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", - "type": "central", - "url": "https://my.zerotier.com/" - }, - } -} -``` - - - - The `Dockerfile` will copy the ZeroTier service as well as the ZeroTier CLI to the image: - -``` -FROM node:4.4 -EXPOSE 8080 -COPY server.js . -COPY zerotier . -COPY zerotier-cli . -COPY entrypoint.sh . -COPY .zerotierCliSettings ?> -CMD node server.js -``` - - - The `entrypoint.sh` script will start the ZeroTier service in the VM, attempt to join your deployment network and automatically authorize the new VM if your network is set to private: - -``` -./zerotier -zerotier-cli join $(NWID).conf -zerotier-cli net-auth $(NWID) $(DEVID) -``` - -**Step 5: Lastly, build the image:** - -`docker build -t gcr.io/$PROJECT_ID/hello-node .` - - - -**Step 6: Push the docker image to your *Container Registry*** - -`gcloud docker push gcr.io/$PROJECT_ID/hello-node:v1` - - -## Deploy! - -**Step 7: Create Kubernetes Cluster** - -`gcloud config set compute/zone us-central1-a` - -`gcloud container clusters create hello-world` - -`gcloud container clusters get-credentials hello-world` - - - -**Step 8: Create your pod** - -`kubectl run hello-node --image=gcr.io/$PROJECT_ID/hello-node:v1 --port=8080` - - - -**Step 9: Scale** - -`kubectl scale deployment hello-node --replicas=4` - -## Verify - -Now, after a minute or so you can use `zerotier-cli net-members ` to show all of your VM instances on your ZeroTier deployment network. If you haven't [configured your local CLI](https://github.com/zerotier/ZeroTierOne/tree/dev/cli), you can simply log into [my.zerotier.com](https://my.zerotier.com), go to *Networks -> nwid* to check that your VMs are indeed members of your private network. -- cgit v1.2.3 From 5bff70194b8c6928e0c828139652fd881dcf3afb Mon Sep 17 00:00:00 2001 From: Adam Ierymenko Date: Fri, 23 Dec 2016 14:30:27 -0800 Subject: Ship pre-build man pages. We stopped doing this for a bit due to nit-picky Debian craziness but since we are not targeting core Debian at the moment bring it back. --- .gitignore | 3 -- doc/zerotier-cli.1 | 83 ++++++++++++++++++++++++++++++++++++++++ doc/zerotier-idtool.1 | 84 ++++++++++++++++++++++++++++++++++++++++ doc/zerotier-one.8 | 104 ++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 271 insertions(+), 3 deletions(-) create mode 100644 doc/zerotier-cli.1 create mode 100644 doc/zerotier-idtool.1 create mode 100644 doc/zerotier-one.8 (limited to 'doc') diff --git a/.gitignore b/.gitignore index d368697c..eb07dd4d 100755 --- a/.gitignore +++ b/.gitignore @@ -69,9 +69,6 @@ zt1-src.tar.gz *.rpm *.autosave *.tmp -doc/*.1 -doc/*.2 -doc/*.8 .depend node_modules debian/files diff --git a/doc/zerotier-cli.1 b/doc/zerotier-cli.1 new file mode 100644 index 00000000..167109ec --- /dev/null +++ b/doc/zerotier-cli.1 @@ -0,0 +1,83 @@ +.TH "ZEROTIER\-CLI" "1" "December 2016" "" "" +.SH "NAME" +\fBzerotier-cli\fR \- control local ZeroTier virtual network service +.SH SYNOPSIS +.P +\fBzerotier\-cli\fP [\-switches] [arguments] +.SH DESCRIPTION +.P +\fBzerotier\-cli\fR provides a simple command line interface to the local JSON API of the ZeroTier virtual network endpoint service zerotier\-one(8)\. +.P +By default \fBzerotier\-cli\fR must be run as root or with \fBsudo\fP\|\. If you want to allow an unprivileged user to use \fBzerotier\-cli\fR to control the system ZeroTier service, you can create a local copy of the ZeroTier service authorization token in the user's home directory: +.P +.RS 2 +.nf +sudo cp /var/lib/zerotier\-one/authtoken\.secret /home/user/\.zeroTierOneAuthToken +chown user /home/user/\.zeroTierOneAuthToken +chmod 0600 /home/user/\.zeroTierOneAuthToken +.fi +.RE +.P +(The location of ZeroTier's service home may differ by platform\. See zerotier\-one(8)\.) +.P +Note that this gives the user the power to connect or disconnect the system to or from any virtual network, which is a significant permission\. +.P +\fBzerotier\-cli\fR has several command line arguments that are visible in \fBhelp\fP output\. The two most commonly used are \fB\-j\fP for raw JSON output and \fB\-D\fP to specify an alternative ZeroTier service working directory\. Raw JSON output is easier to parse in scripts and also contains verbose details not present in the tabular output\. The \fB\-D\fP option specifies where the service's zerotier\-one\.port and authtoken\.secret files are located if the service is not running at the default location for your system\. +.SH COMMANDS +.RS 0 +.IP \(bu 2 +\fBhelp\fP: +Displays \fBzerotier\-cli\fR help\. +.IP \(bu 2 +\fBinfo\fP: +Shows information about this device including its 10\-digit ZeroTier address and apparent connection status\. Use \fB\-j\fP for more verbose output\. +.IP \(bu 2 +\fBlistpeers\fP: +This command lists the ZeroTier VL1 (virtual layer 1, the peer to peer network) peers this service knows about and has recently (within the past 30 minutes or so) communicated with\. These are not necessarily all the devices on your virtual network(s), and may also include a few devices not on any virtual network you've joined\. These are typically either root servers or network controllers\. +.IP \(bu 2 +\fBlistnetworks\fP: +This lists the networks your system belongs to and some information about them, such as any ZeroTier\-managed IP addresses you have been assigned\. (IP addresses assigned manually to ZeroTier interfaces will not be listed here\. Use the standard network interface commands to see these\.) +.IP \(bu 2 +\fBjoin\fP: +To join a network just use \fBjoin\fP and its 16\-digit hex network ID\. That's it\. Then use \fBlistnetworks\fP to see the status\. You'll either get a reply from the network controller with a certificate and other info such as IP assignments, or you'll get "access denied\." In this case you'll need the administrator of this network to authorize your device by its 10\-digit device ID (visible with \fBinfo\fP) on the network's controller\. +.IP \(bu 2 +\fBleave\fP: +Leaving a network is as easy as joining it\. This disconnects from the network and deletes its interface from the system\. Note that peers on the network may hang around in \fBlistpeers\fP for up to 30 minutes until they time out due to lack of traffic\. But if they no longer share a network with you, they can't actually communicate with you in any meaningful way\. + +.RE +.SH EXAMPLES +.P +Join "Earth," ZeroTier's big public party line network: +.P +.RS 2 +.nf +$ sudo zerotier\-cli join 8056c2e21c000001 +$ sudo zerotier\-cli listnetworks +( wait until you get an Earth IP ) +$ ping earth\.zerotier\.net +( you should now be able to ping our Earth test IP ) +.fi +.RE +.P +Leave "Earth": +.P +.RS 2 +.nf +$ sudo zerotier\-cli leave 8056c2e21c000001 +.fi +.RE +.P +List VL1 peers: +.P +.RS 2 +.nf +$ sudo zerotier\-cli listpeers +.fi +.RE +.SH COPYRIGHT +.P +(c)2011\-2016 ZeroTier, Inc\. \-\- https://www\.zerotier\.com/ \-\- https://github\.com/zerotier +.SH SEE ALSO +.P +zerotier\-one(8), zerotier\-idtool(1) + diff --git a/doc/zerotier-idtool.1 b/doc/zerotier-idtool.1 new file mode 100644 index 00000000..fbc367a6 --- /dev/null +++ b/doc/zerotier-idtool.1 @@ -0,0 +1,84 @@ +.TH "ZEROTIER\-IDTOOL" "1" "December 2016" "" "" +.SH "NAME" +\fBzerotier-idtool\fR \- tool for creating and manipulating ZeroTier identities +.SH SYNOPSIS +.P +\fBzerotier\-idtool\fP [args] +.SH DESCRIPTION +.P +\fBzerotier\-idtool\fR is a command line utility for doing things with ZeroTier identities\. A ZeroTier identity consists of a public/private key pair (or just the public if it's only an identity\.public) and a 10\-digit hexadecimal ZeroTier address derived from the public key by way of a proof of work based hash function\. +.SH COMMANDS +.P +When command arguments call for a public or secret (full) identity, the identity can be specified as a path to a file or directly on the command line\. +.RS 0 +.IP \(bu 2 +\fBhelp\fP: +Display help\. (Also running with no command does this\.) +.IP \(bu 2 +\fBgenerate\fP [secret file] [public file] [vanity]: +Generate a new ZeroTier identity\. If a secret file is specified, the full identity including the private key will be written to this file\. If the public file is specified, the public portion will be written there\. If no file paths are specified the full secret identity is output to STDOUT\. The vanity prefix is a series of hexadecimal digits that the generated identity's address should start with\. Typically this isn't used, and if it's specified generation can take a very long time due to the intrinsic cost of generating identities with their proof of work function\. Generating an identity with a known 16\-bit (4 digit) prefix on a 2\.8ghz Core i5 (using one core) takes an average of two hours\. +.IP \(bu 2 +\fBvalidate\fP : +Locally validate an identity's key and proof of work function correspondence\. +.IP \(bu 2 +\fBgetpublic\fP : +Extract the public portion of an identity\.secret and print to STDOUT\. +.IP \(bu 2 +\fBsign\fP : +Sign a file's contents with SHA512+ECC\-256 (ed25519)\. The signature is output in hex to STDOUT\. +.IP \(bu 2 +\fBverify\fP : +Verify a signature created with \fBsign\fP\|\. +.IP \(bu 2 +\fBmkcom\fP [id,value,maxdelta] [\|\.\.\.]: +Create and sign a network membership certificate\. This is not generally useful since network controllers do this automatically and is included mostly for testing purposes\. + +.RE +.SH EXAMPLES +.P +Generate and dump a new identity: +.P +.RS 2 +.nf +$ zerotier\-idtool generate +.fi +.RE +.P +Generate and write a new identity, both secret and public parts: +.P +.RS 2 +.nf +$ zerotier\-idtool generate identity\.secret identity\.public +.fi +.RE +.P +Generate a vanity address that begins with the hex digits "beef" (this will take a while!): +.P +.RS 2 +.nf +$ zerotier\-idtool generate beef\.secret beef\.public beef +.fi +.RE +.P +Sign a file with an identity's secret key: +.P +.RS 2 +.nf +$ zerotier\-idtool sign identity\.secret last_will_and_testament\.txt +.fi +.RE +.P +Verify a file's signature with a public key: +.P +.RS 2 +.nf +$ zerotier\-idtool verify identity\.public last_will_and_testament\.txt +.fi +.RE +.SH COPYRIGHT +.P +(c)2011\-2016 ZeroTier, Inc\. \-\- https://www\.zerotier\.com/ \-\- https://github\.com/zerotier +.SH SEE ALSO +.P +zerotier\-one(8), zerotier\-cli(1) + diff --git a/doc/zerotier-one.8 b/doc/zerotier-one.8 new file mode 100644 index 00000000..4ad7a15d --- /dev/null +++ b/doc/zerotier-one.8 @@ -0,0 +1,104 @@ +.TH "ZEROTIER\-ONE" "8" "December 2016" "" "" +.SH "NAME" +\fBzerotier-one\fR \- ZeroTier virtual network endpoint service +.SH SYNOPSIS +.P +\fBzerotier\-one\fP [\-switches] [working directory] +.SH DESCRIPTION +.P +\fBzerotier\-one\fR is the service/daemon responsible for connecting a Unix (Linux/BSD/OSX) system to one or more ZeroTier virtual networks and presenting those networks to the system as virtual network ports\. You can think of it as a peer to peer VPN client\. +.P +It's typically run by init systems like systemd (Linux) or launchd (Mac) rather than directly by the user, and it must be run as root unless you give it the \fB\-U\fP switch and don't plan on actually joining networks (e\.g\. to run a network controller microservice only)\. +.P +The \fBzerotier\-one\fR service keeps its state and other files in a working directory\. If this directory is not specified at launch it defaults to "/var/lib/zerotier\-one" on Linux, "/Library/Application Support/ZeroTier/One" on Mac, and "/var/db/zerotier\-one" on FreeBSD and other similar BSDs\. The working directory should persist\. It shouldn't be automatically cleaned by system cleanup daemons or stored in a volatile location\. Loss of its identity\.secret file results in loss of this system's unique 10\-digit ZeroTier address and key\. +.P +Multiple instances of \fBzerotier\-one\fR can be run on the same system as long as they are run with different primary ports (see switches) and a different working directory\. But since a single service can join any number of networks, typically there's no point in doing this\. +.P +The \fBzerotier\-one\fR service is controlled via a JSON API available at 127\.0\.0\.1: with the default primary port being 9993\. Access to this API requires an authorization token normally found in the authtoken\.secret file in the service's working directory\. On some platforms access may be guarded by other measures such as socket peer UID/GID lookup if additional security options are enabled (this is not the default)\. +.P +The first time the service is started in a fresh working directory, it generates a ZeroTier identity\. On slow systems this process can take ten seconds or more due to an anti\-DDOS/anti\-counterfeit proof of work function used by ZeroTier in address generation\. This only happens once, and once generated the result is saved in identity\.secret in the working directory\. This file represents and defines/claims your ZeroTier address and associated ECC\-256 key pair\. +.SH SWITCHES +.RS 0 +.IP \(bu 2 +\fB\-h\fP: +Display help\. +.IP \(bu 2 +\fB\-v\fP: +Display ZeroTier One version\. +.IP \(bu 2 +\fB\-U\fP: +Skip privilege check and allow to be run by non\-privileged user\. This is typically used when \fBzerotier\-one\fR is built with the network controller option included\. In this case the ZeroTier service might only be acting as a network controller and might never actually join networks, in which case it does not require elevated system permissions\. +.IP \(bu 2 +\fB\-p\fP: +Specify a different primary port\. If this is not given the default is 9993\. If zero is given a random port is chosen each time\. +.IP \(bu 2 +\fB\-d\fP: +Fork and run as a daemon\. +.IP \(bu 2 +\fB\-i\fP: +Invoke the \fBzerotier\-idtool\fR personality, in which case the binary behaves like zerotier\-idtool(1)\. This happens automatically if the name of the binary (or a symlink to it) is zerotier\-idtool\. +.IP \(bu 2 +\fB\-q\fP: +Invoke the \fBzerotier\-cli\fR personality, in which case the binary behaves like zerotier\-cli(1)\. This happens automatically if the name of the binary (or a symlink to it) is zerotier\-cli\. + +.RE +.SH EXAMPLES +.P +Run as daemon with OS default working directory and default port: +.P +.RS 2 +.nf +$ sudo zerotier\-one \-d +.fi +.RE +.P +Run as daemon with a different working directory and port: +.P +.RS 2 +.nf +$ sudo zerotier\-one \-d \-p12345 /tmp/zerotier\-working\-directory\-test +.fi +.RE +.SH FILES +.P +These are found in the service's working directory\. +.RS 0 +.IP \(bu 2 +\fBidentity\.public\fP: +The public portion of your ZeroTier identity, which is your 10\-digit hex address and the associated public key\. +.IP \(bu 2 +\fBidentity\.secret\fP: +Your full ZeroTier identity including its private key\. This file identifies the system on the network, which means you can move a ZeroTier address around by copying this file and you should back up this file if you want to save your system's static ZeroTier address\. This file must be protected, since theft of its secret key will allow anyone to impersonate your device on any network and decrypt traffic\. For network controllers this file is particularly sensitive since it constitutes the private key for a certificate authority for the controller's networks\. +.IP \(bu 2 +\fBauthtoken\.secret\fP: +The secret token used to authenticate requests to the service's local JSON API\. If it does not exist it is generated from a secure random source on service start\. To use, send it in the "X\-ZT1\-Auth" header with HTTP requests to 127\.0\.0\.1:\|\. +.IP \(bu 2 +\fBdevicemap\fP: +Remembers mappings of zt# interface numbers to ZeroTier networks so they'll persist across restarts\. On some systems that support longer interface names that can encode the network ID (such as FreeBSD) this file may not be present\. +.IP \(bu 2 +\fBzerotier\-one\.pid\fP: +ZeroTier's PID\. This file is deleted on normal shutdown\. +.IP \(bu 2 +\fBzerotier\-one\.port\fP: +ZeroTier's primary port, which is also where its JSON API is found at 127\.0\.0\.1:\|\. This file is created on startup and is read by zerotier\-cli(1) to determine where it should find the control API\. +.IP \(bu 2 +\fBcontroller\.db\fP: +If the ZeroTier One service is built with the network controller enabled, this file contains the controller's SQLite3 database\. +.IP \(bu 2 +\fBcontroller\.db\.backup\fP: +If the ZeroTier One service is built with the network controller enabled, it periodically backs up its controller\.db database in this file (currently every 5 minutes if there have been changes)\. Since this file is not a currently in use SQLite3 database it's safer to back up without corruption\. On new backups the file is rotated out rather than being rewritten in place\. +.IP \(bu 2 +\fBiddb\.d/\fP (directory): +Caches the public identity of every peer ZeroTier has spoken with in the last 60 days\. This directory and its contents can be deleted, but this may result in slower connection initations since it will require that we go out and re\-fetch full identities for peers we're speaking to\. +.IP \(bu 2 +\fBnetworks\.d\fP (directory): +This caches network configurations and certificate information for networks you belong to\. ZeroTier scans this directory for \|\.conf files on startup to recall its networks, so "touch"ing an empty \|\.conf file in this directory is a way of pre\-configuring ZeroTier to join a specific network on startup without using the API\. If the config file is empty ZeroTIer will just fetch it from the network's controller\. + +.RE +.SH COPYRIGHT +.P +(c)2011\-2016 ZeroTier, Inc\. \-\- https://www\.zerotier\.com/ \-\- https://github\.com/zerotier +.SH SEE ALSO +.P +zerotier\-cli(1), zerotier\-idtool(1) + -- cgit v1.2.3 From be2881ae8d00ddf21ab515b636ca41707c8f1cea Mon Sep 17 00:00:00 2001 From: Adam Ierymenko Date: Thu, 2 Feb 2017 16:53:27 -0800 Subject: . --- doc/MANUAL.md | 76 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 doc/MANUAL.md (limited to 'doc') diff --git a/doc/MANUAL.md b/doc/MANUAL.md new file mode 100644 index 00000000..baca77ba --- /dev/null +++ b/doc/MANUAL.md @@ -0,0 +1,76 @@ +ZeroTier Planetary Switch Users Guide +====== + +This manual describes the design and operation of ZeroTier and its associated services, apps, and libraries. Its intended audience includes IT professionals, network administrators, information security experts, and developers. + +[ZeroTier Central](https://my.zerotier.com/), our enterprise web UI, has its own documentation that can be accessed through its interface. The high level ZeroTier rule description language used in Central also has its own documentation. This guide describes the core ZeroTier engine as deployed at endpoints as well as the deployment and operation of network controllers themselves. + +## Table of Contents + +1. [Introduction](#1) +2. [How it Works](#2) + 1. [VL1: The ZeroTier Peer to Peer Network](#2_1) + 1. [ZeroTier Addressing System](#2_1_1) + 2. [Encryption and Authentication](#2_1_2) + 3. [Trust Relationships](#2_1_3) + 4. [Global Roots and WAN Peer Discovery](#2_1_4) + 5. [Direct vs. Relayed Communication](#2_1_5) + 6. [LAN Peer Discovery](#2_1_6) + 7. [Federated Roots ("Moons")](#2_1_7) + 8. [Trusted Paths for Fast Local SDN](#2_1_8) + 9. [Troubleshooting Connectivity Problems](#2_1_9) + 2. [VL2: The Ethernet Virtualization Layer](#2_2) + 1. [Networks Identifiers and Controllers](#2_2_1) + 2. [Multicast](#2_2_2) + 3. [Ethernet Bridging](#2_2_3) + 4. [Certificates of Membership](#2_2_4) + 5. [Revocations and Fast Propagation](#2_2_5) + 6. [Rules Engine](#2_2_6) + 7. [Capabilities](#2_2_7) + 8. [Tags](#2_2_8) +5. [ZeroTier One: The Network Virtualization Service](#5) + 1. [Installation and Operation](#5_1) + 1. [Joining and Leaving Networks](#5_1_1) + 2. [Listing VL1 Peers for Network Troubleshooting](#5_1_2) + 3. [Joining Federated Roots ("Orbiting Moons")](#5_1_3) + 2. [Communicating With the Service](#5_2) + 3. [Operating a Network Controller](#5_3) + 1. [Controller Configuration](#5_3_1) + 2. [High Availability](#5_3_2) + 3. [Security Considerations](#5_3_3) + 4. [Mobile Apps](#5_4) + 5. [Advanced Topics](#5_5) + 1. [Static Paths and Interface Exclusion](#5_5_1) + 2. [Defining Trusted Paths](#5_5_2) + 3. [Allowing Remote Administrative Requests](#5_5_3) + 4. [Creating Federated Roots ("Moons")](#5_5_4) + 5. [Clustering and Geo-Optimized Routing](#5_5_5) +6. [Common Use Cases](#6) + 1. [SDN and General Network Virtualization](#6_1) + 2. [Replacing Conventional VPNs for Remote Access](#6_2) + 1. [Layer 2 Bridge Deployment Strategy](#6_2_1) + 2. [Layer 3 Router Deployment Strategy](#6_2_2) + 3. [Full Tunnel / Default Gateway Override](#6_2_3) + 3. [Multi-Site and Hybrid Cloud Backplane](#6_3) + 1. [Configuring for Specific Cloud Providers](#6_3_1) + 1. [Amazon](#6_3_1_1) + 2. [Microsoft Azure](#6_3_1_2) + 3. [Google Cloud](#6_3_1_3) + 4. [Digital Ocean, Vultr, Linode, OVH, etc.](#6_3_1_4) + 4. [SD-WAN for Site-to-Site Connectivity](#6_4) + 1. [Layer 2 Bridge Deployment Strategy](#6_4_1) + 2. [Layer 3 Router Deployment Strategy](#6_4_2) + 5. [Deivce (IoT) and Application Peer-to-Peer Networking](#6_5) + 1. [Running ZeroTier One in Containers and Virtual Appliances](#6_5_1) + 2. [ZeroTier One on Linux or BSD Powered IoT Devices](#6_5_2) +7. [For Developers: Connecting IoT Devices and Apps](#7) + 1. [ZeroTier SDK for Apps](#7_1) + 2. [ZeroTier Core](#7_2) + 1. [Code Layout and Design](#7_2_1) + 2. [Building and Using](#7_2_2) +7. [Licensing](#8) + +------ + +## **1.** Introduction + -- cgit v1.2.3 From 8a2ff0b31ea5df159b90640ccb9b86e8cb9040bc Mon Sep 17 00:00:00 2001 From: Adam Ierymenko Date: Fri, 3 Feb 2017 19:47:00 -0800 Subject: Actual documentation. --- doc/MANUAL.md | 105 +++++++++++++++++++++++++++++++++++++++++++++++++++------ node/README.md | 2 +- 2 files changed, 95 insertions(+), 12 deletions(-) (limited to 'doc') diff --git a/doc/MANUAL.md b/doc/MANUAL.md index baca77ba..abaeef26 100644 --- a/doc/MANUAL.md +++ b/doc/MANUAL.md @@ -3,22 +3,18 @@ ZeroTier Planetary Switch Users Guide This manual describes the design and operation of ZeroTier and its associated services, apps, and libraries. Its intended audience includes IT professionals, network administrators, information security experts, and developers. -[ZeroTier Central](https://my.zerotier.com/), our enterprise web UI, has its own documentation that can be accessed through its interface. The high level ZeroTier rule description language used in Central also has its own documentation. This guide describes the core ZeroTier engine as deployed at endpoints as well as the deployment and operation of network controllers themselves. +[ZeroTier Central](https://my.zerotier.com/), our enterprise web UI, has its own help and documentation that can be accessed through its interface. ## Table of Contents 1. [Introduction](#1) 2. [How it Works](#2) 1. [VL1: The ZeroTier Peer to Peer Network](#2_1) - 1. [ZeroTier Addressing System](#2_1_1) - 2. [Encryption and Authentication](#2_1_2) - 3. [Trust Relationships](#2_1_3) - 4. [Global Roots and WAN Peer Discovery](#2_1_4) - 5. [Direct vs. Relayed Communication](#2_1_5) - 6. [LAN Peer Discovery](#2_1_6) - 7. [Federated Roots ("Moons")](#2_1_7) - 8. [Trusted Paths for Fast Local SDN](#2_1_8) - 9. [Troubleshooting Connectivity Problems](#2_1_9) + 1. [Network Topology and Peer Discovery](#2_1_1) + 2. [Addressing](#2_1_2) + 3. [Encryption and Authentication](#2_1_3) + 4. [Trusted Paths for Fast Local SDN](#2_1_4) + 5. [Troubleshooting Connectivity Problems](#2_1_5) 2. [VL2: The Ethernet Virtualization Layer](#2_2) 1. [Networks Identifiers and Controllers](#2_2_1) 2. [Multicast](#2_2_2) @@ -65,7 +61,7 @@ This manual describes the design and operation of ZeroTier and its associated se 2. [ZeroTier One on Linux or BSD Powered IoT Devices](#6_5_2) 7. [For Developers: Connecting IoT Devices and Apps](#7) 1. [ZeroTier SDK for Apps](#7_1) - 2. [ZeroTier Core](#7_2) + 2. [ZeroTier Network Hypervisor Core](#7_2) 1. [Code Layout and Design](#7_2_1) 2. [Building and Using](#7_2_2) 7. [Licensing](#8) @@ -74,3 +70,90 @@ This manual describes the design and operation of ZeroTier and its associated se ## **1.** Introduction +ZeroTier is a smart Ethernet switch for planet Earth. + +When the world is a single data center VPN, SDN, SD-WAN, and application peer to peer networking converge. The vast byzantine complexity of managing all these systems independently largely disappears. We've re-thought networking from first principles to deliver the flat end-to-end simplicity of the original pre-NAT pre-mobility Internet but in a way that meets the security, privacy, and mobility requirements of the 21st century. + +This guide is written for users with at least an intermediate understanding of topics like Ethernet and TCP/IP networking. It explains ZeroTier's design and use in considerable detail. Most users with sufficient IT expertise to configure a router or firewall will *not* need this guide to deploy ZeroTier for simple use cases. Indeed we've built a substantial user base prior to its publication. + +So before reading all this you might want to just try installing ZeroTier on a few things and creating a network. Come back when you want to understand what's happening or when you need to make use of more advanced features like rules, capabilities, federation, or clustering. + +## **2.** How it Works + +This section explains how ZeroTier's network hypervisor works. It's not required reading to operate ZeroTier for all but the most advanced deployments, but understanding how things work is always helpful if you ever need to troubleshoot anything. + +ZeroTier is comprised of two closely coupled but conceptually distinct layers [in the OSI model](https://en.wikipedia.org/wiki/OSI_model) sense: a virtual "wire" layer called VL1 that moves data around and a virtual switched Ethernet layer called VL2 to provide devices and apps with a familiar interface. Since almost any protocol can be carried over Ethernet, emulating standard Ethernet behavior maximizes versatility. + +### **2.1.** VL1: The ZeroTier Peer to Peer Network + +To build a planetary data center we first had to begin with the wiring. Tunneling into the Earth's core and putting a giant wire closet down there wasn't an option, so we decided to use software to build virtual wires over the existing Internet instead. + +In conventional networks L1 (OSI layer 1) refers to the actual CAT5/CAT6 cables or wireless radio channels over which data is carried and the physical transciever chips that modulate and demodulate it. VL1 is a peer to peer network that does the same thing by using encryption, authentication, and a lot of networking tricks to create virtual wires as needed. + +### **2.1.1.** Network Topology and Peer Discovery + +VL1's persistent structure is a hierarchical tree similar to DNS, but its leaves make direct ephemeral connections to one another on demand. At the base of the tree resides a pool of equal and fully redundant *roots* whose function is closely analogous to that of [DNS root name servers](https://en.wikipedia.org/wiki/Root_name_server). + +Roots run the same software as regular endpoints but reside at fast stable locations on the network and are designated as such by a *world definition*. There are two kinds of world definitions: a *planet* and a *moon*. The ZeroTier protocol contains a secure mechanism allowing world definitions to be updated in band. + +There is only one planet. Earth's root servers are operated by ZeroTier, Inc. as a free service. Their presence defines and unifies the global data center where we all reside. + +Users can create "moons." These nominate additional roots for redundancy or performance. The most common reasons for doing this are to eliminate hard dependency on ZeroTier's third party infrastructure or to designate local roots inside your building or cloud so ZeroTier can work without a connection to the Internet. Moons are by no means required and most of our users get by just fine without them. + +When peers start out they have no direct links to one another, only upstream to roots. Every peer on VL1 possesses a globally unique address, but unlike IP addresses these are opaque cryptographic identifiers that encode no routing information. To communicate peers first send packets "up" the tree, and as these packets traverse the network they trigger the opportunistic creation of direct links along the way. The tree is constantly trying to "collapse itself" to optimize itself to the pattern of traffic it is carrying. + +In the simplest case using only global roots, an initial connection setup between peers A and B goes like this: + +1. A wants to send a packet to B, but since it has no direct path it sends it upstream to R (a root). +2. R *does* have a direct link to B so it forwards the packet there. +3. R also sends a message called *RENDEZVOUS* to A containing hints about how it might reach B, and to B informing it how it might reach A. We call this "transport triggered link provisioning." +4. A and B get *RENDEZVOUS* and attempt to send test messages to each other, possibly accomplishing [hole punching](https://en.wikipedia.org/wiki/UDP_hole_punching) of any NATs or stateful firewalls that happen to be in the way. If this works a direct link is established and packets no longer need to take the scenic route. + +If R is a federated root the same process occurs, but possibly with additional steps. + +VL1 provides instant always-on virtual L1 connectivity between all devices in the world. Indirect paths are automatically and transparently upgraded to direct paths whenever possible, and if a direct path is lost ZeroTier falls back to indirect communication and the process begins again. + +If a direct path can never be established, indirect communication can continue forever with direct connection attempts also continuing indefinitely on a periodic basis. The protocol also contains other facilities for direct connectivity establishment such as LAN peer discovery, port prediction to traverse IPv4 symmetric NATs, and explicit endpoint advertisement that can be coupled with port mapping using uPnP or NAT-PMP if available. VL1 is persistent and determined when it comes to finding the most efficient way to move data. + +This is not a wholly unique design. It shares features in common with STUN/ICE, SIP, and other protocols. The most novel aspect may be the simplification achieved through lazy transport triggered link provisioning. This trades the [complicated state machines of STUN/ICE](https://www.pkcsecurity.com/untangling-webrtc-flow.html) for a stateless algorithm with implicit empirical parameters. It also eliminates asymmetry. As mentioned above, roots run the same code as regular nodes. + +*[A blog post from 2014 by ZeroTier's original author explains some of the reasoning process that led to this design.](http://adamierymenko.com/decentralization-i-want-to-believe/)* + +### **2.1.2.** Addressing + +Every device (a "device" can be anything from a laptop to an app) is identified on VL1 by a 40-bit (10 hex digit) unique *ZeroTier address*. These are the addresses used to address packets in the process described in 2.1.1 above. + +These addresses are computed from the public portion of a public/private key pair. An address along with its public key is called an *identity*. If you look at the home directory of a running ZeroTier instance you will see `identity.public` and `identity.secret`. + +When ZeroTier starts for the first time it generates a new key pair and a new identity. It then attempts to advertise it upstream to the network. In the very unlikely event that the identity's 40-bit unique address is taken, it discards it and generates another. + +Identities are claimed on a first come first serve basis and currently expire from global roots after 60 days of inactivity. If a long-dormant device returns it may re-claim its identity unless its address has been taken in the meantime (again, highly unlikely). + +The address derivation algorithm used to compute addresses from public keys imposes a computational cost barrier against the intentional generation of a collision. Currently it would take approximately 10,000 CPU-years to do so (assuming e.g. a 3ghz Intel core). This is expensive but not impossible, but it's only the first line of defense. After generating a collision an attacker would then have to compromise all upstream nodes and replace the address's cached identity, not to mention also doing the same for peers that have seen the target identity recently. + +In addition to assisting with communication, upstream nodes also act as identity caches. If the identity corresponding to an address is not known a peer may request it by sending a message called *WHOIS* upstream. + +### **2.1.3.** Encryption and Authentication + +If you don't know much about cryptography you can safely skip this section. **TL;DR: packets are end-to-end encrypted and can't be read by roots or anyone else, and we use modern 256-bit crypto in ways recommended by the professional cryptographers that created it.** + +Asymmetric public key encryption is [Curve25519/Ed25519](https://en.wikipedia.org/wiki/Curve25519), a 256-bit elliptic curve variant. + +Every VL1 packet is encrypted end to end using (as of the current version) 256-bit [Salsa20](https://ianix.com/pub/salsa20-deployment.html) and authenticated using the [Poly1305](https://en.wikipedia.org/wiki/Poly1305) message authentication (MAC) algorithm. + +MAC is computed after encryption [(encrypt-then-MAC)](https://tonyarcieri.com/all-the-crypto-code-youve-ever-written-is-probably-broken) and the cipher/MAC composition used is identical to the [NaCl reference implementation](https://nacl.cr.yp.to). + +As of today we do not implement [forward secrecy](https://en.wikipedia.org/wiki/Forward_secrecy) or other stateful cryptographic features in VL1. We don't do this for the sake of simplicity, reliability, and code footprint, and because frequently changing state makes features like clustering and fail-over much harder to implement. See [our discussion on GitHub](https://github.com/zerotier/ZeroTierOne/issues/204). + +For those who have very high security needs and want forward secrecy, we currently recommend the use of encrypted protocols such as SSH and SSL over ZeroTier. Not only do these provide forward secrecy, the use of multiple layers of encryption in this way provides excellent [defense in depth](https://en.wikipedia.org/wiki/Defense_in_depth_%28computing%29). The computational cost of this additional crypto is typically small, and the benefit can potentially be large. All software can contain bugs, but multiple layers of protection means that discovery of a catastrophic bug in any one layer does not result in compromise of your entire system. We recommend the same for authentication. While ZeroTier VL2 provides certificate-based network boundary enforcement, we do not recommend that users rely solely on this for access control to critical systems. It is always good to use more than one security measure whenever practical. + +### **2.1.4.** Trusted Paths for Fast Local SDN + +To support the use of ZeroTier as a high performance SDN/NFV protocol over physically secure networks the protocol supports a feature called *trusted paths*. It is possible to configure all ZeroTier devices on a given network to skip encryption and authentication for traffic over a designated physical path. This can cut CPU use noticably in high traffic scenarios but at the expense of effectively all transport security over the configured trusted backplane. + +Trusted paths do not prevent communication with devices elsewhere, since traffic over other paths will be encrypted and authenticated normally. + +We don't recommend the use of this feature unless you really need the performance and you know what you're doing. Extra security is never a bad thing. We also recommend thinking carefully before disabling transport security on a cloud private network. Larger cloud providers such as Amazon and Azure tend to provide good network segregation but many less costly providers offer private networks that are "party lines." For these the encryption and authentication provided by ZeroTier is very desirable. In fact, we have a few users using ZeroTier exactly for this reason. + +### **2.1.5.** Troubleshooting Connectivity Problems + diff --git a/node/README.md b/node/README.md index 01378c75..1728400c 100644 --- a/node/README.md +++ b/node/README.md @@ -1,4 +1,4 @@ -ZeroTier Virtual Switch Core +ZeroTier Network Hypervisor Core ====== This directory contains the *real* ZeroTier: a completely OS-independent global virtual Ethernet switch engine. This is where the magic happens. -- cgit v1.2.3 From 31db768e4d4c2815d2be0493b2c76ea5f5edbffa Mon Sep 17 00:00:00 2001 From: Adam Ierymenko Date: Sat, 4 Feb 2017 00:23:31 -0800 Subject: A bit of code cleanup. --- controller/EmbeddedNetworkController.cpp | 2 +- doc/MANUAL.md | 2 - node/Address.hpp | 77 +++++++++----------------------- node/Identity.cpp | 2 +- node/NetworkConfig.cpp | 2 +- node/Switch.cpp | 3 +- service/OneService.cpp | 2 +- 7 files changed, 25 insertions(+), 65 deletions(-) (limited to 'doc') diff --git a/controller/EmbeddedNetworkController.cpp b/controller/EmbeddedNetworkController.cpp index 3c13e7ce..e7bcdd07 100644 --- a/controller/EmbeddedNetworkController.cpp +++ b/controller/EmbeddedNetworkController.cpp @@ -1690,7 +1690,7 @@ void EmbeddedNetworkController::_getNetworkMemberInfo(uint64_t now,uint64_t nwid } if (OSUtils::jsonBool(member["activeBridge"],false)) { - nmi.activeBridges.insert(OSUtils::jsonString(member["id"],"0000000000")); + nmi.activeBridges.insert(Address(Utils::hexStrToU64(OSUtils::jsonString(member["id"],"0000000000").c_str()))); } if (member.count("ipAssignments")) { diff --git a/doc/MANUAL.md b/doc/MANUAL.md index abaeef26..8af8fc2d 100644 --- a/doc/MANUAL.md +++ b/doc/MANUAL.md @@ -109,8 +109,6 @@ In the simplest case using only global roots, an initial connection setup betwee 3. R also sends a message called *RENDEZVOUS* to A containing hints about how it might reach B, and to B informing it how it might reach A. We call this "transport triggered link provisioning." 4. A and B get *RENDEZVOUS* and attempt to send test messages to each other, possibly accomplishing [hole punching](https://en.wikipedia.org/wiki/UDP_hole_punching) of any NATs or stateful firewalls that happen to be in the way. If this works a direct link is established and packets no longer need to take the scenic route. -If R is a federated root the same process occurs, but possibly with additional steps. - VL1 provides instant always-on virtual L1 connectivity between all devices in the world. Indirect paths are automatically and transparently upgraded to direct paths whenever possible, and if a direct path is lost ZeroTier falls back to indirect communication and the process begins again. If a direct path can never be established, indirect communication can continue forever with direct connection attempts also continuing indefinitely on a periodic basis. The protocol also contains other facilities for direct connectivity establishment such as LAN peer discovery, port prediction to traverse IPv4 symmetric NATs, and explicit endpoint advertisement that can be coupled with port mapping using uPnP or NAT-PMP if available. VL1 is persistent and determined when it comes to finding the most efficient way to move data. diff --git a/node/Address.hpp b/node/Address.hpp index 9bf5605a..4a5883b0 100644 --- a/node/Address.hpp +++ b/node/Address.hpp @@ -38,57 +38,26 @@ namespace ZeroTier { class Address { public: - Address() - throw() : - _a(0) - { - } - - Address(const Address &a) - throw() : - _a(a._a) - { - } - - Address(uint64_t a) - throw() : - _a(a & 0xffffffffffULL) - { - } - - Address(const char *s) - throw() - { - unsigned char foo[ZT_ADDRESS_LENGTH]; - setTo(foo,Utils::unhex(s,foo,ZT_ADDRESS_LENGTH)); - } - - Address(const std::string &s) - throw() - { - unsigned char foo[ZT_ADDRESS_LENGTH]; - setTo(foo,Utils::unhex(s.c_str(),foo,ZT_ADDRESS_LENGTH)); - } + Address() : _a(0) {} + Address(const Address &a) : _a(a._a) {} + Address(uint64_t a) : _a(a & 0xffffffffffULL) {} /** * @param bits Raw address -- 5 bytes, big-endian byte order * @param len Length of array */ Address(const void *bits,unsigned int len) - throw() { setTo(bits,len); } inline Address &operator=(const Address &a) - throw() { _a = a._a; return *this; } inline Address &operator=(const uint64_t a) - throw() { _a = (a & 0xffffffffffULL); return *this; @@ -99,7 +68,6 @@ public: * @param len Length of array */ inline void setTo(const void *bits,unsigned int len) - throw() { if (len < ZT_ADDRESS_LENGTH) { _a = 0; @@ -119,7 +87,6 @@ public: * @param len Length of array */ inline void copyTo(void *bits,unsigned int len) const - throw() { if (len < ZT_ADDRESS_LENGTH) return; @@ -138,7 +105,6 @@ public: */ template inline void appendTo(Buffer &b) const - throw(std::out_of_range) { unsigned char *p = (unsigned char *)b.appendField(ZT_ADDRESS_LENGTH); *(p++) = (unsigned char)((_a >> 32) & 0xff); @@ -152,7 +118,6 @@ public: * @return Integer containing address (0 to 2^40) */ inline uint64_t toInt() const - throw() { return _a; } @@ -161,7 +126,6 @@ public: * @return Hash code for use with Hashtable */ inline unsigned long hashCode() const - throw() { return (unsigned long)_a; } @@ -188,12 +152,12 @@ public: /** * @return True if this address is not zero */ - inline operator bool() const throw() { return (_a != 0); } + inline operator bool() const { return (_a != 0); } /** * Set to null/zero */ - inline void zero() throw() { _a = 0; } + inline void zero() { _a = 0; } /** * Check if this address is reserved @@ -205,7 +169,6 @@ public: * @return True if address is reserved and may not be used */ inline bool isReserved() const - throw() { return ((!_a)||((_a >> 32) == ZT_ADDRESS_RESERVED_PREFIX)); } @@ -214,21 +177,21 @@ public: * @param i Value from 0 to 4 (inclusive) * @return Byte at said position (address interpreted in big-endian order) */ - inline unsigned char operator[](unsigned int i) const throw() { return (unsigned char)((_a >> (32 - (i * 8))) & 0xff); } - - inline bool operator==(const uint64_t &a) const throw() { return (_a == (a & 0xffffffffffULL)); } - inline bool operator!=(const uint64_t &a) const throw() { return (_a != (a & 0xffffffffffULL)); } - inline bool operator>(const uint64_t &a) const throw() { return (_a > (a & 0xffffffffffULL)); } - inline bool operator<(const uint64_t &a) const throw() { return (_a < (a & 0xffffffffffULL)); } - inline bool operator>=(const uint64_t &a) const throw() { return (_a >= (a & 0xffffffffffULL)); } - inline bool operator<=(const uint64_t &a) const throw() { return (_a <= (a & 0xffffffffffULL)); } - - inline bool operator==(const Address &a) const throw() { return (_a == a._a); } - inline bool operator!=(const Address &a) const throw() { return (_a != a._a); } - inline bool operator>(const Address &a) const throw() { return (_a > a._a); } - inline bool operator<(const Address &a) const throw() { return (_a < a._a); } - inline bool operator>=(const Address &a) const throw() { return (_a >= a._a); } - inline bool operator<=(const Address &a) const throw() { return (_a <= a._a); } + inline unsigned char operator[](unsigned int i) const { return (unsigned char)((_a >> (32 - (i * 8))) & 0xff); } + + inline bool operator==(const uint64_t &a) const { return (_a == (a & 0xffffffffffULL)); } + inline bool operator!=(const uint64_t &a) const { return (_a != (a & 0xffffffffffULL)); } + inline bool operator>(const uint64_t &a) const { return (_a > (a & 0xffffffffffULL)); } + inline bool operator<(const uint64_t &a) const { return (_a < (a & 0xffffffffffULL)); } + inline bool operator>=(const uint64_t &a) const { return (_a >= (a & 0xffffffffffULL)); } + inline bool operator<=(const uint64_t &a) const { return (_a <= (a & 0xffffffffffULL)); } + + inline bool operator==(const Address &a) const { return (_a == a._a); } + inline bool operator!=(const Address &a) const { return (_a != a._a); } + inline bool operator>(const Address &a) const { return (_a > a._a); } + inline bool operator<(const Address &a) const { return (_a < a._a); } + inline bool operator>=(const Address &a) const { return (_a >= a._a); } + inline bool operator<=(const Address &a) const { return (_a <= a._a); } private: uint64_t _a; diff --git a/node/Identity.cpp b/node/Identity.cpp index c47805d9..05b70873 100644 --- a/node/Identity.cpp +++ b/node/Identity.cpp @@ -160,7 +160,7 @@ bool Identity::fromString(const char *str) for(char *f=Utils::stok(tmp,":",&saveptr);(f);f=Utils::stok((char *)0,":",&saveptr)) { switch(fno++) { case 0: - _address = Address(f); + _address = Address(Utils::hexStrToU64(f)); if (_address.isReserved()) return false; break; diff --git a/node/NetworkConfig.cpp b/node/NetworkConfig.cpp index 6acc48ea..2f356b15 100644 --- a/node/NetworkConfig.cpp +++ b/node/NetworkConfig.cpp @@ -258,7 +258,7 @@ bool NetworkConfig::fromDictionary(const Dictionary 0) { char *saveptr = (char *)0; for(char *f=Utils::stok(tmp2,",",&saveptr);(f);f=Utils::stok((char *)0,",",&saveptr)) { - this->addSpecialist(Address(f),ZT_NETWORKCONFIG_SPECIALIST_TYPE_ACTIVE_BRIDGE); + this->addSpecialist(Address(Utils::hexStrToU64(f)),ZT_NETWORKCONFIG_SPECIALIST_TYPE_ACTIVE_BRIDGE); } } #else diff --git a/node/Switch.cpp b/node/Switch.cpp index 9fcd379f..a769faea 100644 --- a/node/Switch.cpp +++ b/node/Switch.cpp @@ -224,7 +224,7 @@ void Switch::onRemotePacket(const InetAddress &localAddr,const InetAddress &from #endif SharedPtr relayTo = RR->topology->getPeer(destination); - if ((relayTo)&&((relayTo->sendDirect(packet.data(),packet.size(),now,false)))) { + if ((relayTo)&&(relayTo->sendDirect(packet.data(),packet.size(),now,false))) { if ((source != RR->identity.address())&&(_shouldUnite(now,source,destination))) { // don't send RENDEZVOUS for cluster frontplane relays const InetAddress *hintToSource = (InetAddress *)0; const InetAddress *hintToDest = (InetAddress *)0; @@ -245,7 +245,6 @@ void Switch::onRemotePacket(const InetAddress &localAddr,const InetAddress &from } if ((hintToSource)&&(hintToDest)) { - TRACE(">> RENDEZVOUS: %s(%s) <> %s(%s)",p1.toString().c_str(),p1a->toString().c_str(),p2.toString().c_str(),p2a->toString().c_str()); unsigned int alt = (unsigned int)RR->node->prng() & 1; // randomize which hint we send first for obscure NAT-t reasons const unsigned int completed = alt + 2; while (alt != completed) { diff --git a/service/OneService.cpp b/service/OneService.cpp index 49c5f4a0..9a1503e5 100644 --- a/service/OneService.cpp +++ b/service/OneService.cpp @@ -926,7 +926,7 @@ public: for(json::iterator v(virt.begin());v!=virt.end();++v) { const std::string nstr = v.key(); if ((nstr.length() == ZT_ADDRESS_LENGTH_HEX)&&(v.value().is_object())) { - const Address ztaddr(nstr.c_str()); + const Address ztaddr(Utils::hexStrToU64(nstr.c_str())); if (ztaddr) { const uint64_t ztaddr2 = ztaddr.toInt(); std::vector &v4h = _v4Hints[ztaddr2]; -- cgit v1.2.3 From 5eab6c2e5db4c7a1a99aa32cd18c1512a1a5173f Mon Sep 17 00:00:00 2001 From: Adam Ierymenko Date: Sat, 4 Feb 2017 19:19:30 -0800 Subject: More docs. --- doc/MANUAL.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) (limited to 'doc') diff --git a/doc/MANUAL.md b/doc/MANUAL.md index 8af8fc2d..e84146f3 100644 --- a/doc/MANUAL.md +++ b/doc/MANUAL.md @@ -24,7 +24,7 @@ This manual describes the design and operation of ZeroTier and its associated se 6. [Rules Engine](#2_2_6) 7. [Capabilities](#2_2_7) 8. [Tags](#2_2_8) -5. [ZeroTier One: The Network Virtualization Service](#5) +3. [ZeroTier One: The Network Virtualization Service](#5) 1. [Installation and Operation](#5_1) 1. [Joining and Leaving Networks](#5_1_1) 2. [Listing VL1 Peers for Network Troubleshooting](#5_1_2) @@ -41,7 +41,7 @@ This manual describes the design and operation of ZeroTier and its associated se 3. [Allowing Remote Administrative Requests](#5_5_3) 4. [Creating Federated Roots ("Moons")](#5_5_4) 5. [Clustering and Geo-Optimized Routing](#5_5_5) -6. [Common Use Cases](#6) +4. [Common Use Cases](#6) 1. [SDN and General Network Virtualization](#6_1) 2. [Replacing Conventional VPNs for Remote Access](#6_2) 1. [Layer 2 Bridge Deployment Strategy](#6_2_1) @@ -59,12 +59,12 @@ This manual describes the design and operation of ZeroTier and its associated se 5. [Deivce (IoT) and Application Peer-to-Peer Networking](#6_5) 1. [Running ZeroTier One in Containers and Virtual Appliances](#6_5_1) 2. [ZeroTier One on Linux or BSD Powered IoT Devices](#6_5_2) -7. [For Developers: Connecting IoT Devices and Apps](#7) +5. [For Developers: Connecting IoT Devices and Apps](#7) 1. [ZeroTier SDK for Apps](#7_1) 2. [ZeroTier Network Hypervisor Core](#7_2) 1. [Code Layout and Design](#7_2_1) 2. [Building and Using](#7_2_2) -7. [Licensing](#8) +6. [Licensing](#8) ------ @@ -72,17 +72,17 @@ This manual describes the design and operation of ZeroTier and its associated se ZeroTier is a smart Ethernet switch for planet Earth. -When the world is a single data center VPN, SDN, SD-WAN, and application peer to peer networking converge. The vast byzantine complexity of managing all these systems independently largely disappears. We've re-thought networking from first principles to deliver the flat end-to-end simplicity of the original pre-NAT pre-mobility Internet but in a way that meets the security, privacy, and mobility requirements of the 21st century. +We've re-thought networking from first principles to deliver the flat end-to-end simplicity of the original pre-NAT pre-mobility Internet in a way that meets the security and mobility requirements of the 21st century. ZeroTier transforms the world into a unified modern data center where VPN, SDN, SD-WAN, and application peer to peer networking converge and where the distinction between the cloud and the endpoint largely disappears. All the complexity of managing these networking aspects as disparate systems is replaced by the simplicity of a single virtual cloud. -This guide is written for users with at least an intermediate understanding of topics like Ethernet and TCP/IP networking. It explains ZeroTier's design and use in considerable detail. Most users with sufficient IT expertise to configure a router or firewall will *not* need this guide to deploy ZeroTier for simple use cases. Indeed we've built a substantial user base prior to its publication. +At first some users struggle with this paradigm, finding it difficult to forget the fragmentation and complexity that has accreted around networking over the past decade or two. We urge skeptical users to just try it and see how many networking acronyms vanish before their eyes. -So before reading all this you might want to just try installing ZeroTier on a few things and creating a network. Come back when you want to understand what's happening or when you need to make use of more advanced features like rules, capabilities, federation, or clustering. +Unlike most networking products it won't take you hours, days, or weeks to test or deploy ZeroTier. Most of the time everything just works with zero configuration, and most users with some level of TCP/IP knowledge can get up and running in minutes. More advanced features like rules, micro-segmentation, capability based security credentials, network monitoring, and clustering are available but you don't need to worry about them until they're needed. -## **2.** How it Works +The first section (2) of this guide explains ZeroTier's design and operation in detail and is written for users with at least an intermediate knowledge of topics like TCP/IP and Ethernet networking. Reading and understanding everything in it is not mandatory but we've written it as a deep technical dive as serious IT users typically like to understand the systems they deploy and use. Sections 3 and 4 deal more concretely with the ZeroTier One endpoint service software and how to deploy for common use cases. -This section explains how ZeroTier's network hypervisor works. It's not required reading to operate ZeroTier for all but the most advanced deployments, but understanding how things work is always helpful if you ever need to troubleshoot anything. +## **2.** How it Works -ZeroTier is comprised of two closely coupled but conceptually distinct layers [in the OSI model](https://en.wikipedia.org/wiki/OSI_model) sense: a virtual "wire" layer called VL1 that moves data around and a virtual switched Ethernet layer called VL2 to provide devices and apps with a familiar interface. Since almost any protocol can be carried over Ethernet, emulating standard Ethernet behavior maximizes versatility. +ZeroTier is comprised of two closely coupled but conceptually distinct layers [in the OSI model](https://en.wikipedia.org/wiki/OSI_model) sense: a virtual "wire" layer called VL1 that carries data and a virtual switched Ethernet layer called VL2 to provide devices and apps with a familiar communication paradigm. ### **2.1.** VL1: The ZeroTier Peer to Peer Network -- cgit v1.2.3 From 43182f8f57483a47f1b44cdcf9dbb5387511afc2 Mon Sep 17 00:00:00 2001 From: Adam Ierymenko Date: Sun, 5 Feb 2017 16:19:03 -0800 Subject: Docs, code cleanup, and protect the extra new fields of HELLO with encryption as a precaution. --- doc/MANUAL.md | 12 +++---- node/Constants.hpp | 5 +++ node/Identity.cpp | 6 ++-- node/IncomingPacket.cpp | 88 +++++++++++++++++++++++++++++-------------------- node/Node.cpp | 4 +-- node/Packet.cpp | 62 +++++++++++++++++++++++----------- node/Packet.hpp | 31 ++++++++++++++--- node/Peer.cpp | 22 ++++++++----- node/Peer.hpp | 3 +- node/Salsa20.cpp | 4 +-- node/Salsa20.hpp | 34 +++---------------- node/Switch.cpp | 2 +- node/Utils.cpp | 4 +-- 13 files changed, 162 insertions(+), 115 deletions(-) (limited to 'doc') diff --git a/doc/MANUAL.md b/doc/MANUAL.md index e84146f3..e0117a4d 100644 --- a/doc/MANUAL.md +++ b/doc/MANUAL.md @@ -72,13 +72,11 @@ This manual describes the design and operation of ZeroTier and its associated se ZeroTier is a smart Ethernet switch for planet Earth. -We've re-thought networking from first principles to deliver the flat end-to-end simplicity of the original pre-NAT pre-mobility Internet in a way that meets the security and mobility requirements of the 21st century. ZeroTier transforms the world into a unified modern data center where VPN, SDN, SD-WAN, and application peer to peer networking converge and where the distinction between the cloud and the endpoint largely disappears. All the complexity of managing these networking aspects as disparate systems is replaced by the simplicity of a single virtual cloud. +We've re-thought networking from first principles to deliver the flat end-to-end simplicity of the original pre-NAT pre-mobility Internet, yet in a way that meets the security and mobility needs of the 21st century. ZeroTier erases the distinction between cloud and endpoint, bringing modern data center SDN to every device regardless of its physical location. The objectives of VPN, SDN, SD-WAN, and application peer to peer networking can all be achieved together and with reduced complexity. -At first some users struggle with this paradigm, finding it difficult to forget the fragmentation and complexity that has accreted around networking over the past decade or two. We urge skeptical users to just try it and see how many networking acronyms vanish before their eyes. +Unlike most networking products it won't take you hours, days, or weeks to test or deploy ZeroTier. Most of the time everything just works with zero configuration, and most users with some level of TCP/IP knowledge can get up and running in minutes. More advanced features like federation, rules, micro-segmentation, capability based security credentials, network monitoring, and clustering are available but you don't need to worry about them until they're needed. -Unlike most networking products it won't take you hours, days, or weeks to test or deploy ZeroTier. Most of the time everything just works with zero configuration, and most users with some level of TCP/IP knowledge can get up and running in minutes. More advanced features like rules, micro-segmentation, capability based security credentials, network monitoring, and clustering are available but you don't need to worry about them until they're needed. - -The first section (2) of this guide explains ZeroTier's design and operation in detail and is written for users with at least an intermediate knowledge of topics like TCP/IP and Ethernet networking. Reading and understanding everything in it is not mandatory but we've written it as a deep technical dive as serious IT users typically like to understand the systems they deploy and use. Sections 3 and 4 deal more concretely with the ZeroTier One endpoint service software and how to deploy for common use cases. +The first section (2) of this guide explains ZeroTier's design and operation in detail and is written for users with at least an intermediate knowledge of topics like TCP/IP and Ethernet networking. All of it is not required reading for most users, but we've created a deep technical dive to satisfy the desire of IT professionals to understand the systems that they use. Sections 3 and 4 deal more concretely with the ZeroTier One endpoint service software and how to deploy for common use cases. ## **2.** How it Works @@ -88,7 +86,7 @@ ZeroTier is comprised of two closely coupled but conceptually distinct layers [i To build a planetary data center we first had to begin with the wiring. Tunneling into the Earth's core and putting a giant wire closet down there wasn't an option, so we decided to use software to build virtual wires over the existing Internet instead. -In conventional networks L1 (OSI layer 1) refers to the actual CAT5/CAT6 cables or wireless radio channels over which data is carried and the physical transciever chips that modulate and demodulate it. VL1 is a peer to peer network that does the same thing by using encryption, authentication, and a lot of networking tricks to create virtual wires as needed. +In conventional networks L1 (OSI layer 1) refers to the actual CAT5/CAT6 cables or wireless radio channels over which data is carried and the physical transciever chips that modulate and demodulate it. VL1 is a peer to peer network that does the same thing by using encryption, authentication, and a lot of networking tricks to create virtual wires on a dyniamic as-needed basis. ### **2.1.1.** Network Topology and Peer Discovery @@ -98,7 +96,7 @@ Roots run the same software as regular endpoints but reside at fast stable locat There is only one planet. Earth's root servers are operated by ZeroTier, Inc. as a free service. Their presence defines and unifies the global data center where we all reside. -Users can create "moons." These nominate additional roots for redundancy or performance. The most common reasons for doing this are to eliminate hard dependency on ZeroTier's third party infrastructure or to designate local roots inside your building or cloud so ZeroTier can work without a connection to the Internet. Moons are by no means required and most of our users get by just fine without them. +Moons can be created by users. These nominate additional roots for redundancy or performance. The most common reasons for doing this are to eliminate hard dependency on ZeroTier's third party infrastructure or to designate local roots inside your building or cloud so ZeroTier can work without a connection to the Internet. Moons are by no means required and most of our users get by just fine without them. When peers start out they have no direct links to one another, only upstream to roots. Every peer on VL1 possesses a globally unique address, but unlike IP addresses these are opaque cryptographic identifiers that encode no routing information. To communicate peers first send packets "up" the tree, and as these packets traverse the network they trigger the opportunistic creation of direct links along the way. The tree is constantly trying to "collapse itself" to optimize itself to the pattern of traffic it is carrying. diff --git a/node/Constants.hpp b/node/Constants.hpp index be4eb475..3bda3805 100644 --- a/node/Constants.hpp +++ b/node/Constants.hpp @@ -293,6 +293,11 @@ */ #define ZT_PEER_PATH_EXPIRATION ((ZT_PEER_PING_PERIOD * 4) + 3000) +/** + * Send a full HELLO every this often (ms) + */ +#define ZT_PEER_SEND_FULL_HELLO_EVERY (ZT_PEER_PING_PERIOD * 2) + /** * How often to retry expired paths that we're still remembering */ diff --git a/node/Identity.cpp b/node/Identity.cpp index 05b70873..89fdb836 100644 --- a/node/Identity.cpp +++ b/node/Identity.cpp @@ -46,7 +46,7 @@ static inline void _computeMemoryHardHash(const void *publicKey,unsigned int pub // but is not what we want for sequential memory-harndess. memset(genmem,0,ZT_IDENTITY_GEN_MEMORY); Salsa20 s20(digest,256,(char *)digest + 32); - s20.encrypt20((char *)genmem,(char *)genmem,64); + s20.crypt20((char *)genmem,(char *)genmem,64); for(unsigned long i=64;i(ZT_PROTO_VERB_HELLO_IDX_REVISION); const uint64_t timestamp = at(ZT_PROTO_VERB_HELLO_IDX_TIMESTAMP); - Identity id; - InetAddress externalSurfaceAddress; - uint64_t planetWorldId = 0; - uint64_t planetWorldTimestamp = 0; - std::vector< std::pair > moonIdsAndTimestamps; - { - unsigned int ptr = ZT_PROTO_VERB_HELLO_IDX_IDENTITY + id.deserialize(*this,ZT_PROTO_VERB_HELLO_IDX_IDENTITY); - - // Get external surface address if present (was not in old versions) - if (ptr < size()) - ptr += externalSurfaceAddress.deserialize(*this,ptr); - - // Get primary planet world ID and world timestamp if present - if ((ptr + 16) <= size()) { - planetWorldId = at(ptr); ptr += 8; - planetWorldTimestamp = at(ptr); - } - - // Get moon IDs and timestamps if present - if ((ptr + 2) <= size()) { - unsigned int numMoons = at(ptr); ptr += 2; - for(unsigned int i=0;i(at(ptr),at(ptr + 8))); - ptr += 16; - } - } + if (protoVersion < ZT_PROTO_VERSION_MIN) { + TRACE("dropped HELLO from %s(%s): protocol version too old",id.address().toString().c_str(),_path->address().toString().c_str()); + return true; } + Identity id; + unsigned int ptr = ZT_PROTO_VERB_HELLO_IDX_IDENTITY + id.deserialize(*this,ZT_PROTO_VERB_HELLO_IDX_IDENTITY); + if (fromAddress != id.address()) { - TRACE("dropped HELLO from %s(%s): identity not for sending address",fromAddress.toString().c_str(),_path->address().toString().c_str()); - return true; - } - if (protoVersion < ZT_PROTO_VERSION_MIN) { - TRACE("dropped HELLO from %s(%s): protocol version too old",id.address().toString().c_str(),_path->address().toString().c_str()); + TRACE("dropped HELLO from %s(%s): identity does not match packet source address",fromAddress.toString().c_str(),_path->address().toString().c_str()); return true; } @@ -324,6 +299,43 @@ bool IncomingPacket::_doHELLO(const RuntimeEnvironment *RR,const bool alreadyAut // VALID -- if we made it here, packet passed identity and authenticity checks! + // Get external surface address if present (was not in old versions) + InetAddress externalSurfaceAddress; + if (ptr < size()) + ptr += externalSurfaceAddress.deserialize(*this,ptr); + + // Get primary planet world ID and world timestamp if present + uint64_t planetWorldId = 0; + uint64_t planetWorldTimestamp = 0; + if ((ptr + 16) <= size()) { + planetWorldId = at(ptr); ptr += 8; + planetWorldTimestamp = at(ptr); + } + + std::vector< std::pair > moonIdsAndTimestamps; + if (ptr < size()) { + // Remainder of packet, if present, is encrypted + cryptField(peer->key(),ptr,size() - ptr); + + // Get moon IDs and timestamps if present + if ((ptr + 2) <= size()) { + unsigned int numMoons = at(ptr); ptr += 2; + for(unsigned int i=0;i(at(ptr),at(ptr + 8))); + ptr += 16; + } + } + + // Handle COR if present (older versions don't send this) + if ((ptr + 2) <= size()) { + //const unsigned int corSize = at(ptr); ptr += 2; + ptr += 2; + CertificateOfRepresentation cor; + ptr += cor.deserialize(*this,ptr); + } + } + // Learn our external surface address from other peers to help us negotiate symmetric NATs // and detect changes to our global IP that can trigger path renegotiation. if ((externalSurfaceAddress)&&(hops() == 0)) @@ -337,6 +349,7 @@ bool IncomingPacket::_doHELLO(const RuntimeEnvironment *RR,const bool alreadyAut outp.append((unsigned char)ZEROTIER_ONE_VERSION_MAJOR); outp.append((unsigned char)ZEROTIER_ONE_VERSION_MINOR); outp.append((uint16_t)ZEROTIER_ONE_VERSION_REVISION); + if (protoVersion >= 5) { _path->address().serialize(outp); } else { @@ -387,6 +400,11 @@ bool IncomingPacket::_doHELLO(const RuntimeEnvironment *RR,const bool alreadyAut } outp.setAt(worldUpdateSizeAt,(uint16_t)(outp.size() - (worldUpdateSizeAt + 2))); + const unsigned int corSizeAt = outp.size(); + outp.addSize(2); + RR->topology->appendCertificateOfRepresentation(outp); + outp.setAt(corSizeAt,(uint16_t)(outp.size() - (corSizeAt + 2))); + outp.armor(peer->key(),true); _path->send(RR,outp.data(),outp.size(),now); @@ -586,7 +604,7 @@ bool IncomingPacket::_doRENDEZVOUS(const RuntimeEnvironment *RR,const SharedPtr< const InetAddress atAddr(field(ZT_PROTO_VERB_RENDEZVOUS_IDX_ADDRESS,addrlen),addrlen,port); if (RR->node->shouldUsePathForZeroTierTraffic(with,_path->localAddress(),atAddr)) { RR->node->putPacket(_path->localAddress(),atAddr,"ABRE",4,2); // send low-TTL junk packet to 'open' local NAT(s) and stateful firewalls - rendezvousWith->attemptToContactAt(_path->localAddress(),atAddr,RR->node->now()); + rendezvousWith->attemptToContactAt(_path->localAddress(),atAddr,RR->node->now(),false); TRACE("RENDEZVOUS from %s says %s might be at %s, sent verification attempt",peer->address().toString().c_str(),with.toString().c_str(),atAddr.toString().c_str()); } else { TRACE("RENDEZVOUS from %s says %s might be at %s, ignoring since path is not suitable",peer->address().toString().c_str(),with.toString().c_str(),atAddr.toString().c_str()); @@ -1155,7 +1173,7 @@ bool IncomingPacket::_doPUSH_DIRECT_PATHS(const RuntimeEnvironment *RR,const Sha if ( ((flags & ZT_PUSH_DIRECT_PATHS_FLAG_FORGET_PATH) == 0) && (!redundant) && (RR->node->shouldUsePathForZeroTierTraffic(peer->address(),_path->localAddress(),a)) ) { if (++countPerScope[(int)a.ipScope()][0] <= ZT_PUSH_DIRECT_PATHS_MAX_PER_SCOPE_AND_FAMILY) { TRACE("attempting to contact %s at pushed direct path %s",peer->address().toString().c_str(),a.toString().c_str()); - peer->attemptToContactAt(InetAddress(),a,now); + peer->attemptToContactAt(InetAddress(),a,now,false); } else { TRACE("ignoring contact for %s at %s -- too many per scope",peer->address().toString().c_str(),a.toString().c_str()); } @@ -1174,7 +1192,7 @@ bool IncomingPacket::_doPUSH_DIRECT_PATHS(const RuntimeEnvironment *RR,const Sha if ( ((flags & ZT_PUSH_DIRECT_PATHS_FLAG_FORGET_PATH) == 0) && (!redundant) && (RR->node->shouldUsePathForZeroTierTraffic(peer->address(),_path->localAddress(),a)) ) { if (++countPerScope[(int)a.ipScope()][1] <= ZT_PUSH_DIRECT_PATHS_MAX_PER_SCOPE_AND_FAMILY) { TRACE("attempting to contact %s at pushed direct path %s",peer->address().toString().c_str(),a.toString().c_str()); - peer->attemptToContactAt(InetAddress(),a,now); + peer->attemptToContactAt(InetAddress(),a,now,false); } else { TRACE("ignoring contact for %s at %s -- too many per scope",peer->address().toString().c_str(),a.toString().c_str()); } diff --git a/node/Node.cpp b/node/Node.cpp index 3d5b5c3d..b8e74a52 100644 --- a/node/Node.cpp +++ b/node/Node.cpp @@ -70,7 +70,7 @@ Node::Node(void *uptr,const struct ZT_Node_Callbacks *callbacks,uint64_t now) : Utils::getSecureRandom(foo,32); _prng.init(foo,256,foo); memset(_prngStream,0,sizeof(_prngStream)); - _prng.encrypt12(_prngStream,_prngStream,sizeof(_prngStream)); + _prng.crypt12(_prngStream,_prngStream,sizeof(_prngStream)); std::string idtmp(dataStoreGet("identity.secret")); if ((!idtmp.length())||(!RR->identity.fromString(idtmp))||(!RR->identity.hasPrivate())) { @@ -686,7 +686,7 @@ uint64_t Node::prng() { unsigned int p = (++_prngStreamPtr % ZT_NODE_PRNG_BUF_SIZE); if (!p) - _prng.encrypt12(_prngStream,_prngStream,sizeof(_prngStream)); + _prng.crypt12(_prngStream,_prngStream,sizeof(_prngStream)); return _prngStream[p]; } diff --git a/node/Packet.cpp b/node/Packet.cpp index 05fe8dd9..a1bb3132 100644 --- a/node/Packet.cpp +++ b/node/Packet.cpp @@ -18,9 +18,21 @@ #include #include +#include +#include +#include #include "Packet.hpp" +#ifdef _MSC_VER +#define FORCE_INLINE static __forceinline +#include +#pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */ +#pragma warning(disable : 4293) /* disable: C4293: too large shift (32-bits) */ +#else +#define FORCE_INLINE static inline +#endif + namespace ZeroTier { /************************************************************************** */ @@ -367,7 +379,7 @@ LZ4_decompress_*_continue() : #define LZ4_HASH_SIZE_U32 (1 << LZ4_HASHLOG) /* required as macro for static allocation */ #if defined(__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) -#include +//#include typedef struct { uint32_t hashTable[LZ4_HASH_SIZE_U32]; @@ -536,6 +548,7 @@ union LZ4_streamDecode_u { /*-************************************ * Compiler Options **************************************/ +#if 0 #ifdef _MSC_VER /* Visual Studio */ # define FORCE_INLINE static __forceinline # include @@ -550,6 +563,7 @@ union LZ4_streamDecode_u { # define FORCE_INLINE static # endif #endif /* _MSC_VER */ +#endif #if (defined(__GNUC__) && (__GNUC__ >= 3)) || (defined(__INTEL_COMPILER) && (__INTEL_COMPILER >= 800)) || defined(__clang__) # define expect(expr,value) (__builtin_expect ((expr),(value)) ) @@ -564,38 +578,39 @@ union LZ4_streamDecode_u { /*-************************************ * Memory routines **************************************/ -#include /* malloc, calloc, free */ +//#include /* malloc, calloc, free */ #define ALLOCATOR(n,s) calloc(n,s) #define FREEMEM free -#include /* memset, memcpy */ +//#include /* memset, memcpy */ #define MEM_INIT memset /*-************************************ * Basic Types **************************************/ -#if defined(__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) -# include +//#if defined(__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) +//# include typedef uint8_t BYTE; typedef uint16_t U16; typedef uint32_t U32; typedef int32_t S32; typedef uint64_t U64; typedef uintptr_t uptrval; -#else +/*#else typedef unsigned char BYTE; typedef unsigned short U16; typedef unsigned int U32; typedef signed int S32; typedef unsigned long long U64; - typedef size_t uptrval; /* generally true, except OpenVMS-64 */ -#endif + typedef size_t uptrval; +#endif */ -#if defined(__x86_64__) - typedef U64 reg_t; /* 64-bits in x32 mode */ -#else - typedef size_t reg_t; /* 32-bits in x32 mode */ -#endif +typedef uintptr_t reg_t; +//#if defined(__x86_64__) +// typedef U64 reg_t; /* 64-bits in x32 mode */ +//#else +// typedef size_t reg_t; /* 32-bits in x32 mode */ +//#endif /*-************************************ * Reading and writing into memory @@ -606,7 +621,6 @@ static unsigned LZ4_isLittleEndian(void) return one.c[0]; } - #if defined(LZ4_FORCE_MEMORY_ACCESS) && (LZ4_FORCE_MEMORY_ACCESS==2) /* lie to the compiler about data alignment; use with caution */ @@ -1975,10 +1989,10 @@ void Packet::armor(const void *key,bool encryptPayload) // MAC key is always the first 32 bytes of the Salsa20 key stream // This is the same construction DJB's NaCl library uses - s20.encrypt12(ZERO_KEY,macKey,sizeof(macKey)); + s20.crypt12(ZERO_KEY,macKey,sizeof(macKey)); if (encryptPayload) - s20.encrypt12(payload,payload,payloadLen); + s20.crypt12(payload,payload,payloadLen); Poly1305::compute(mac,payload,payloadLen,macKey); memcpy(field(ZT_PACKET_IDX_MAC,8),mac,8); @@ -1995,20 +2009,30 @@ bool Packet::dearmor(const void *key) if ((cs == ZT_PROTO_CIPHER_SUITE__C25519_POLY1305_NONE)||(cs == ZT_PROTO_CIPHER_SUITE__C25519_POLY1305_SALSA2012)) { _salsa20MangleKey((const unsigned char *)key,mangledKey); - Salsa20 s20(mangledKey,256,field(ZT_PACKET_IDX_IV,8)/*,ZT_PROTO_SALSA20_ROUNDS*/); + Salsa20 s20(mangledKey,256,field(ZT_PACKET_IDX_IV,8)); - s20.encrypt12(ZERO_KEY,macKey,sizeof(macKey)); + s20.crypt12(ZERO_KEY,macKey,sizeof(macKey)); Poly1305::compute(mac,payload,payloadLen,macKey); if (!Utils::secureEq(mac,field(ZT_PACKET_IDX_MAC,8),8)) return false; if (cs == ZT_PROTO_CIPHER_SUITE__C25519_POLY1305_SALSA2012) - s20.decrypt12(payload,payload,payloadLen); + s20.crypt12(payload,payload,payloadLen); return true; } else return false; // unrecognized cipher suite } +void Packet::cryptField(const void *key,unsigned int start,unsigned int len) +{ + unsigned char mangledKey[32]; + uint64_t iv = Utils::hton((uint64_t)start ^ at(ZT_PACKET_IDX_IV)); + _salsa20MangleKey((const unsigned char *)key,mangledKey); + Salsa20 s20(mangledKey,256,&iv); + unsigned char *const ptr = field(start,len); + s20.crypt12(ptr,ptr,len); +} + bool Packet::compress() { unsigned char buf[ZT_PROTO_MAX_PACKET_LENGTH * 2]; diff --git a/node/Packet.hpp b/node/Packet.hpp index 7d404b25..03bd9ed3 100644 --- a/node/Packet.hpp +++ b/node/Packet.hpp @@ -542,6 +542,7 @@ public: * [<[...] destination address to which packet was sent>] * <[8] 64-bit world ID of current planet> * <[8] 64-bit timestamp of current planet> + * [... remainder if packet is encrypted using cryptField() ...] * <[2] 16-bit number of moons> * [<[1] 8-bit type ID of moon>] * [<[8] 64-bit world ID of moon>] @@ -550,9 +551,10 @@ public: * <[2] 16-bit length of certificate of representation> * [... certificate of representation ...] * - * HELLO is sent in the clear, and therefore cannot contain anything - * secret or highly confidential. It should contain nothing that is - * not either public or easy to obtain via other means. + * The initial fields of HELLO are sent in the clear. Fields after the + * planet definition (which are common knowledge) are however encrypted + * using the cryptField() function. The packet is MAC'd as usual using + * the same MAC construct as other packets. * * The destination address is the wire address to which this packet is * being sent, and in OK is *also* the destination address of the OK @@ -566,7 +568,7 @@ public: * 0x04 - 6-byte IPv4 UDP address/port -- format: <[4] IP>, <[2] port> * 0x06 - 18-byte IPv6 UDP address/port -- format: <[16] IP>, <[2] port> * - * OK payload: + * OK payload (note that OK is encrypted): * <[8] timestamp (echoed from original HELLO)> * <[1] protocol version (of responder)> * <[1] software major version (of responder)> @@ -576,6 +578,8 @@ public: * [<[...] destination address>] * <[2] 16-bit length of world update or 0 if none> * [[...] updates to planets and/or moons] + * <[2] 16-bit length of certificate of representation (of responder)> + * [... certificate of representation ...] * * ERROR has no payload. */ @@ -1348,6 +1352,25 @@ public: */ bool dearmor(const void *key); + /** + * Encrypt/decrypt a separately armored portion of a packet + * + * This keys using the same key in the same way as armor/dearmor, but + * uses a different IV computed from the packet's IV plus the starting + * point index. + * + * This currently uses Salsa20/12, but any message that uses this should + * incorporate a cipher selector to permit this to be changed later. + * + * This is currently only used to mask portions of HELLO as an extra + * security precation since most of that message is sent in the clear. + * + * @param key 32-byte key + * @param start Start of encrypted portion + * @param len Length of encrypted portion + */ + void cryptField(const void *key,unsigned int start,unsigned int len); + /** * Attempt to compress payload if not already (must be unencrypted) * diff --git a/node/Peer.cpp b/node/Peer.cpp index bb6b945d..338bea10 100644 --- a/node/Peer.cpp +++ b/node/Peer.cpp @@ -203,7 +203,7 @@ void Peer::received( #endif } else { TRACE("got %s via unknown path %s(%s), confirming...",Packet::verbString(verb),_id.address().toString().c_str(),path->address().toString().c_str()); - attemptToContactAt(path->localAddress(),path->address(),now); + attemptToContactAt(path->localAddress(),path->address(),now,true); path->sent(now); } } @@ -357,6 +357,8 @@ void Peer::sendHELLO(const InetAddress &localAddr,const InetAddress &atAddress,u outp.append((uint64_t)RR->topology->planetWorldId()); outp.append((uint64_t)RR->topology->planetWorldTimestamp()); + const unsigned int startCryptedPortionAt = outp.size(); + std::vector moons(RR->topology->moons()); outp.append((uint16_t)moons.size()); for(std::vector::const_iterator m(moons.begin());m!=moons.end();++m) { @@ -368,21 +370,23 @@ void Peer::sendHELLO(const InetAddress &localAddr,const InetAddress &atAddress,u const unsigned int corSizeAt = outp.size(); outp.addSize(2); RR->topology->appendCertificateOfRepresentation(outp); - outp.setAt(corSizeAt,(uint16_t)((outp.size() - corSizeAt) - 2)); + outp.setAt(corSizeAt,(uint16_t)(outp.size() - (corSizeAt + 2))); + + outp.cryptField(_key,startCryptedPortionAt,outp.size() - startCryptedPortionAt); RR->node->expectReplyTo(outp.packetId()); if (atAddress) { - outp.armor(_key,false); + outp.armor(_key,false); // false == don't encrypt full payload, but add MAC RR->node->putPacket(localAddr,atAddress,outp.data(),outp.size()); } else { - RR->sw->send(outp,false); + RR->sw->send(outp,false); // false == don't encrypt full payload, but add MAC } } -void Peer::attemptToContactAt(const InetAddress &localAddr,const InetAddress &atAddress,uint64_t now) +void Peer::attemptToContactAt(const InetAddress &localAddr,const InetAddress &atAddress,uint64_t now,bool sendFullHello) { - if ( (_vProto >= 5) && ( !((_vMajor == 1)&&(_vMinor == 1)&&(_vRevision == 0)) ) ) { + if ( (!sendFullHello) && (_vProto >= 5) && (!((_vMajor == 1)&&(_vMinor == 1)&&(_vRevision == 0))) ) { Packet outp(_id.address(),RR->identity.address(),Packet::VERB_ECHO); RR->node->expectReplyTo(outp.packetId()); outp.armor(_key,true); @@ -398,7 +402,7 @@ void Peer::tryMemorizedPath(uint64_t now) _lastTriedMemorizedPath = now; InetAddress mp; if (RR->node->externalPathLookup(_id.address(),-1,mp)) - attemptToContactAt(InetAddress(),mp,now); + attemptToContactAt(InetAddress(),mp,now,true); } } @@ -420,7 +424,7 @@ bool Peer::doPingAndKeepalive(uint64_t now,int inetAddressFamily) if (bestp >= 0) { if ( ((now - _paths[bestp].lastReceive) >= ZT_PEER_PING_PERIOD) || (_paths[bestp].path->needsHeartbeat(now)) ) { - attemptToContactAt(_paths[bestp].path->localAddress(),_paths[bestp].path->address(),now); + attemptToContactAt(_paths[bestp].path->localAddress(),_paths[bestp].path->address(),now,false); _paths[bestp].path->sent(now); } return true; @@ -444,7 +448,7 @@ void Peer::resetWithinScope(InetAddress::IpScope scope,int inetAddressFamily,uin Mutex::Lock _l(_paths_m); for(unsigned int p=0;p<_numPaths;++p) { if ( (_paths[p].path->address().ss_family == inetAddressFamily) && (_paths[p].path->address().ipScope() == scope) ) { - attemptToContactAt(_paths[p].path->localAddress(),_paths[p].path->address(),now); + attemptToContactAt(_paths[p].path->localAddress(),_paths[p].path->address(),now,false); _paths[p].path->sent(now); _paths[p].lastReceive = 0; // path will not be used unless it speaks again } diff --git a/node/Peer.hpp b/node/Peer.hpp index e79739a3..a3ec0088 100644 --- a/node/Peer.hpp +++ b/node/Peer.hpp @@ -161,8 +161,9 @@ public: * @param localAddr Local address * @param atAddress Destination address * @param now Current time + * @param sendFullHello If true, always send a full HELLO instead of just an ECHO */ - void attemptToContactAt(const InetAddress &localAddr,const InetAddress &atAddress,uint64_t now); + void attemptToContactAt(const InetAddress &localAddr,const InetAddress &atAddress,uint64_t now,bool sendFullHello); /** * Try a memorized or statically defined path if any are known diff --git a/node/Salsa20.cpp b/node/Salsa20.cpp index 3aa19ac6..1a4641f7 100644 --- a/node/Salsa20.cpp +++ b/node/Salsa20.cpp @@ -123,7 +123,7 @@ void Salsa20::init(const void *key,unsigned int kbits,const void *iv) #endif } -void Salsa20::encrypt12(const void *in,void *out,unsigned int bytes) +void Salsa20::crypt12(const void *in,void *out,unsigned int bytes) throw() { uint8_t tmp[64]; @@ -623,7 +623,7 @@ void Salsa20::encrypt12(const void *in,void *out,unsigned int bytes) } } -void Salsa20::encrypt20(const void *in,void *out,unsigned int bytes) +void Salsa20::crypt20(const void *in,void *out,unsigned int bytes) throw() { uint8_t tmp[64]; diff --git a/node/Salsa20.hpp b/node/Salsa20.hpp index 7e4c1e53..6405d450 100644 --- a/node/Salsa20.hpp +++ b/node/Salsa20.hpp @@ -56,51 +56,25 @@ public: throw(); /** - * Encrypt data using Salsa20/12 + * Encrypt/decrypt data using Salsa20/12 * * @param in Input data * @param out Output buffer * @param bytes Length of data */ - void encrypt12(const void *in,void *out,unsigned int bytes) + void crypt12(const void *in,void *out,unsigned int bytes) throw(); /** - * Encrypt data using Salsa20/20 + * Encrypt/decrypt data using Salsa20/20 * * @param in Input data * @param out Output buffer * @param bytes Length of data */ - void encrypt20(const void *in,void *out,unsigned int bytes) + void crypt20(const void *in,void *out,unsigned int bytes) throw(); - /** - * Decrypt data - * - * @param in Input data - * @param out Output buffer - * @param bytes Length of data - */ - inline void decrypt12(const void *in,void *out,unsigned int bytes) - throw() - { - encrypt12(in,out,bytes); - } - - /** - * Decrypt data - * - * @param in Input data - * @param out Output buffer - * @param bytes Length of data - */ - inline void decrypt20(const void *in,void *out,unsigned int bytes) - throw() - { - encrypt20(in,out,bytes); - } - private: union { #ifdef ZT_SALSA20_SSE diff --git a/node/Switch.cpp b/node/Switch.cpp index a769faea..346091a4 100644 --- a/node/Switch.cpp +++ b/node/Switch.cpp @@ -777,7 +777,7 @@ bool Switch::_trySend(Packet &packet,bool encrypt) if ((clusterMostRecentMemberId < 0)||(viaPath->lastIn() > clusterMostRecentTs)) { #endif if ((now - viaPath->lastOut()) > std::max((now - viaPath->lastIn()) * 4,(uint64_t)ZT_PATH_MIN_REACTIVATE_INTERVAL)) { - peer->attemptToContactAt(viaPath->localAddress(),viaPath->address(),now); + peer->attemptToContactAt(viaPath->localAddress(),viaPath->address(),now,false); viaPath->sent(now); } #ifdef ZT_ENABLE_CLUSTER diff --git a/node/Utils.cpp b/node/Utils.cpp index 06b726cc..247dd54a 100644 --- a/node/Utils.cpp +++ b/node/Utils.cpp @@ -182,7 +182,7 @@ void Utils::getSecureRandom(void *buf,unsigned int bytes) #else // not __WINDOWS__ - static char randomBuf[131072]; + static char randomBuf[65536]; static unsigned int randomPtr = sizeof(randomBuf); static int devURandomFd = -1; @@ -215,7 +215,7 @@ void Utils::getSecureRandom(void *buf,unsigned int bytes) #endif // __WINDOWS__ or not - s20.encrypt12(buf,buf,bytes); + s20.crypt12(buf,buf,bytes); } bool Utils::scopy(char *dest,unsigned int len,const char *src) -- cgit v1.2.3 From dd92f1d03d85d16fe727780ca56bdaa21c128771 Mon Sep 17 00:00:00 2001 From: Adam Ierymenko Date: Thu, 23 Feb 2017 17:56:36 -0800 Subject: . --- doc/MANUAL.md | 403 +++++++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 301 insertions(+), 102 deletions(-) (limited to 'doc') diff --git a/doc/MANUAL.md b/doc/MANUAL.md index e0117a4d..b26ad3a5 100644 --- a/doc/MANUAL.md +++ b/doc/MANUAL.md @@ -1,155 +1,354 @@ -ZeroTier Planetary Switch Users Guide +ZeroTier Manual ====== -This manual describes the design and operation of ZeroTier and its associated services, apps, and libraries. Its intended audience includes IT professionals, network administrators, information security experts, and developers. +## **1.** Introduction -[ZeroTier Central](https://my.zerotier.com/), our enterprise web UI, has its own help and documentation that can be accessed through its interface. +ZeroTier is a smart Ethernet switch for planet Earth. -## Table of Contents +It's a distributed network hypervisor built atop a cryptographically secure global peer to peer network that provides capabilities similar to an enterprise smart switch such as VLANs, rules, device authentication, and security monitoring. -1. [Introduction](#1) -2. [How it Works](#2) - 1. [VL1: The ZeroTier Peer to Peer Network](#2_1) - 1. [Network Topology and Peer Discovery](#2_1_1) - 2. [Addressing](#2_1_2) - 3. [Encryption and Authentication](#2_1_3) - 4. [Trusted Paths for Fast Local SDN](#2_1_4) - 5. [Troubleshooting Connectivity Problems](#2_1_5) - 2. [VL2: The Ethernet Virtualization Layer](#2_2) - 1. [Networks Identifiers and Controllers](#2_2_1) - 2. [Multicast](#2_2_2) - 3. [Ethernet Bridging](#2_2_3) - 4. [Certificates of Membership](#2_2_4) - 5. [Revocations and Fast Propagation](#2_2_5) - 6. [Rules Engine](#2_2_6) - 7. [Capabilities](#2_2_7) - 8. [Tags](#2_2_8) -3. [ZeroTier One: The Network Virtualization Service](#5) - 1. [Installation and Operation](#5_1) - 1. [Joining and Leaving Networks](#5_1_1) - 2. [Listing VL1 Peers for Network Troubleshooting](#5_1_2) - 3. [Joining Federated Roots ("Orbiting Moons")](#5_1_3) - 2. [Communicating With the Service](#5_2) - 3. [Operating a Network Controller](#5_3) - 1. [Controller Configuration](#5_3_1) - 2. [High Availability](#5_3_2) - 3. [Security Considerations](#5_3_3) - 4. [Mobile Apps](#5_4) - 5. [Advanced Topics](#5_5) - 1. [Static Paths and Interface Exclusion](#5_5_1) - 2. [Defining Trusted Paths](#5_5_2) - 3. [Allowing Remote Administrative Requests](#5_5_3) - 4. [Creating Federated Roots ("Moons")](#5_5_4) - 5. [Clustering and Geo-Optimized Routing](#5_5_5) -4. [Common Use Cases](#6) - 1. [SDN and General Network Virtualization](#6_1) - 2. [Replacing Conventional VPNs for Remote Access](#6_2) - 1. [Layer 2 Bridge Deployment Strategy](#6_2_1) - 2. [Layer 3 Router Deployment Strategy](#6_2_2) - 3. [Full Tunnel / Default Gateway Override](#6_2_3) - 3. [Multi-Site and Hybrid Cloud Backplane](#6_3) - 1. [Configuring for Specific Cloud Providers](#6_3_1) - 1. [Amazon](#6_3_1_1) - 2. [Microsoft Azure](#6_3_1_2) - 3. [Google Cloud](#6_3_1_3) - 4. [Digital Ocean, Vultr, Linode, OVH, etc.](#6_3_1_4) - 4. [SD-WAN for Site-to-Site Connectivity](#6_4) - 1. [Layer 2 Bridge Deployment Strategy](#6_4_1) - 2. [Layer 3 Router Deployment Strategy](#6_4_2) - 5. [Deivce (IoT) and Application Peer-to-Peer Networking](#6_5) - 1. [Running ZeroTier One in Containers and Virtual Appliances](#6_5_1) - 2. [ZeroTier One on Linux or BSD Powered IoT Devices](#6_5_2) -5. [For Developers: Connecting IoT Devices and Apps](#7) - 1. [ZeroTier SDK for Apps](#7_1) - 2. [ZeroTier Network Hypervisor Core](#7_2) - 1. [Code Layout and Design](#7_2_1) - 2. [Building and Using](#7_2_2) -6. [Licensing](#8) +This manual describes the design and operation of ZeroTier and its associated services, apps, and libraries. Its intended audience includes IT professionals, network administrators, information security experts, and developers. ------- +The first section (2) of this guide explains ZeroTier's design and operation at a high level and is written for those with at least an intermediate knowledge of topics like TCP/IP and Ethernet networking. It's not required reading for most users, but understanding how things work in detail helps clarify everything else and helps tremendously with troubleshooting should anything go wrong. -## **1.** Introduction +The remaining sections deal more concretely with deployment and administration. -ZeroTier is a smart Ethernet switch for planet Earth. +### Table of Contents -We've re-thought networking from first principles to deliver the flat end-to-end simplicity of the original pre-NAT pre-mobility Internet, yet in a way that meets the security and mobility needs of the 21st century. ZeroTier erases the distinction between cloud and endpoint, bringing modern data center SDN to every device regardless of its physical location. The objectives of VPN, SDN, SD-WAN, and application peer to peer networking can all be achieved together and with reduced complexity. +1. [Introduction](#1) +2. [Network Hypervisor Overview](#2) + 1. [VL1: The ZeroTier Peer to Peer Network](#2_1) + 1. [Network Topology and Peer Discovery](#2_1_1) + 2. [Addressing](#2_1_2) + 3. [Cryptography](#2_1_3) + 4. [Trusted Paths for Fast Local SDN](#2_1_4) + 2. [VL2: The Ethernet Virtualization Layer](#2_2) + 1. [Network Identifiers and Controllers](#2_2_1) + 2. [Certificates and Other Credentials](#2_2_2) + 3. [Multicast, ARP, NDP, and Special Addressing Modes](#2_2_3) + 4. [Ethernet Bridging](#2_2_4) + 5. [Public Networks](#2_2_5) + 6. [Ad-Hoc Networks](#2_2_6) +3. [The Network Rules Engine](#3) + 1. [Rule Sets and Rule Evaluation](#3_1) + 2. [Capabilities](#3_2) + 3. [Tags](#3_3) + 4. [Rule Description Language](#3_4) + 1. [Syntax](#3_4_1) + 2. [Actions](#3_4_2) + 3. [Match Conditions](#3_4_3) + 4. [Capabilities](#3_4_4) + 5. [Tags](#3_4_5) + 6. [Macros](#3_4_6) + 5. [Design Patterns](#3_5) + 1. [TCP Whitelisting](#3_5_1) + 2. [Low-Overhead Network Monitoring](#3_5_2) -Unlike most networking products it won't take you hours, days, or weeks to test or deploy ZeroTier. Most of the time everything just works with zero configuration, and most users with some level of TCP/IP knowledge can get up and running in minutes. More advanced features like federation, rules, micro-segmentation, capability based security credentials, network monitoring, and clustering are available but you don't need to worry about them until they're needed. +------ -The first section (2) of this guide explains ZeroTier's design and operation in detail and is written for users with at least an intermediate knowledge of topics like TCP/IP and Ethernet networking. All of it is not required reading for most users, but we've created a deep technical dive to satisfy the desire of IT professionals to understand the systems that they use. Sections 3 and 4 deal more concretely with the ZeroTier One endpoint service software and how to deploy for common use cases. +## **2.** Network Hypervisor Overview -## **2.** How it Works +The ZeroTier network hypervisor (currently found in the [node/](https://github.com/zerotier/ZeroTierOne/tree/master/node) subfolder of the ZeroTierOne git repository) is a self-contained network virtualization engine that implements an Ethernet virtualization layer similar to [VXLAN](https://en.wikipedia.org/wiki/Virtual_Extensible_LAN) on top of a global encrypted peer to peer network. -ZeroTier is comprised of two closely coupled but conceptually distinct layers [in the OSI model](https://en.wikipedia.org/wiki/OSI_model) sense: a virtual "wire" layer called VL1 that carries data and a virtual switched Ethernet layer called VL2 to provide devices and apps with a familiar communication paradigm. +The ZeroTier protocol is original, though aspects of it are similar to VXLAN and IPSec. It has two conceptually separate but closely coupled layers [in the OSI model](https://en.wikipedia.org/wiki/OSI_model) sense: **VL1** and **VL2**. VL1 is the underlying peer to peer transport layer, the "virtual wire," while VL2 is an emulated Ethernet layer that provides operating systems and apps with a familiar communication medium. -### **2.1.** VL1: The ZeroTier Peer to Peer Network +### **2.1.** VL1: The ZeroTier Peer to Peer Network -To build a planetary data center we first had to begin with the wiring. Tunneling into the Earth's core and putting a giant wire closet down there wasn't an option, so we decided to use software to build virtual wires over the existing Internet instead. +A global data center requires a global wire closet. In conventional networks L1 (OSI layer 1) refers to the actual CAT5/CAT6 cables or wireless radio channels over which data is carried and the physical transciever chips that modulate and demodulate it. VL1 is a peer to peer network that does the same thing by using encryption, authentication, and a lot of networking tricks to create virtual wires on a dyniamic as-needed basis. -### **2.1.1.** Network Topology and Peer Discovery +### **2.1.1.** Network Topology and Peer Discovery -VL1's persistent structure is a hierarchical tree similar to DNS, but its leaves make direct ephemeral connections to one another on demand. At the base of the tree resides a pool of equal and fully redundant *roots* whose function is closely analogous to that of [DNS root name servers](https://en.wikipedia.org/wiki/Root_name_server). +VL1 is designed to be zero-configuration. A user can start a new ZeroTier node without having to write configuration files or provide the IP addresses of other nodes. It's also designed to be fast. Any two devices in the world should be able to locate each other and communicate almost instantly. -Roots run the same software as regular endpoints but reside at fast stable locations on the network and are designated as such by a *world definition*. There are two kinds of world definitions: a *planet* and a *moon*. The ZeroTier protocol contains a secure mechanism allowing world definitions to be updated in band. +To achieve this VL1 is organized like DNS. At the base of the network is a collection of always-present **root servers** whose role is similar to that of [DNS root name servers](https://en.wikipedia.org/wiki/Root_name_server). Roots run the same software as regular endpoints but reside at fast stable locations on the network and are designated as such by a **world definition**. World definitions come in two forms: the **planet** and one or more **moons**. The protocol includes a secure mechanism allowing world definitions to be updated in-band if root servers' IP addresses or ZeroTier addresses change. -There is only one planet. Earth's root servers are operated by ZeroTier, Inc. as a free service. Their presence defines and unifies the global data center where we all reside. +There is only one planet. Earth's root servers are operated by ZeroTier, Inc. as a free service. There are currently twelve root servers organized into two six-member clusters distributed across every major continent and multiple network providers. Almost everyone in the world has one within less than 100ms network latency from their location. -Moons can be created by users. These nominate additional roots for redundancy or performance. The most common reasons for doing this are to eliminate hard dependency on ZeroTier's third party infrastructure or to designate local roots inside your building or cloud so ZeroTier can work without a connection to the Internet. Moons are by no means required and most of our users get by just fine without them. +A node can "orbit" any number of moons. A moon is just a convenient way to add user-defined root servers to the pool. Users can create moons to reduce dependency on ZeroTier, Inc. infrastructure or to locate root servers closer for better performance. For on-premise SDN use a cluster of root servers can be located inside a building or data center so that ZeroTier can continue to operate normally if Internet connectivity is lost. -When peers start out they have no direct links to one another, only upstream to roots. Every peer on VL1 possesses a globally unique address, but unlike IP addresses these are opaque cryptographic identifiers that encode no routing information. To communicate peers first send packets "up" the tree, and as these packets traverse the network they trigger the opportunistic creation of direct links along the way. The tree is constantly trying to "collapse itself" to optimize itself to the pattern of traffic it is carrying. +Nodes start with no direct links to one another, only upstream to roots (planet and moons). Every peer on VL1 possesses a globally unique 40-bit (10 hex digit) **ZeroTier address**, but unlike IP addresses these are opaque cryptographic identifiers that encode no routing information. To communicate peers first send packets "up" the tree, and as these packets traverse the network they trigger the opportunistic creation of direct links along the way. The tree is constantly trying to "collapse itself" to optimize itself to the pattern of traffic it is carrying. -In the simplest case using only global roots, an initial connection setup between peers A and B goes like this: +Peer to peer connection setup goes like this: 1. A wants to send a packet to B, but since it has no direct path it sends it upstream to R (a root). -2. R *does* have a direct link to B so it forwards the packet there. -3. R also sends a message called *RENDEZVOUS* to A containing hints about how it might reach B, and to B informing it how it might reach A. We call this "transport triggered link provisioning." -4. A and B get *RENDEZVOUS* and attempt to send test messages to each other, possibly accomplishing [hole punching](https://en.wikipedia.org/wiki/UDP_hole_punching) of any NATs or stateful firewalls that happen to be in the way. If this works a direct link is established and packets no longer need to take the scenic route. +2. If R has a direct link to B, it forwards the packet there. Otherwise it sends the packet upstream until planetary roots are reached. Planetary roots know about all nodes, so eventually the packet will reach B if B is online. +3. R also sends a message called *rendezvous* to A containing hints about how it might reach B. Meanwhile the root that forwards the packet to B sends *rendezvous* informing B how it might reach A. +4. A and B get their *rendezvous* messages and attempt to send test messages to each other, possibly accomplishing [hole punching](https://en.wikipedia.org/wiki/UDP_hole_punching) of any NATs or stateful firewalls that happen to be in the way. If this works a direct link is established and packets no longer need to take the scenic route. -VL1 provides instant always-on virtual L1 connectivity between all devices in the world. Indirect paths are automatically and transparently upgraded to direct paths whenever possible, and if a direct path is lost ZeroTier falls back to indirect communication and the process begins again. +Since roots forward packets, A and B can reach each other instantly. A and B then begin attempting to make a direct peer to peer connection. If this succeeds it results in a faster lower latency link. We call this *transport triggered link provisioning* since it's the forwarding of the packet itself that triggers the peer to peer network to attempt direct connection. -If a direct path can never be established, indirect communication can continue forever with direct connection attempts also continuing indefinitely on a periodic basis. The protocol also contains other facilities for direct connectivity establishment such as LAN peer discovery, port prediction to traverse IPv4 symmetric NATs, and explicit endpoint advertisement that can be coupled with port mapping using uPnP or NAT-PMP if available. VL1 is persistent and determined when it comes to finding the most efficient way to move data. +VL1 never gives up. If a direct path can't be established, communication can continue through (slower) relaying. Direct connection attempts continue forever on a periodic basis. VL1 also has other features for establishing direct connectivity including LAN peer discovery, port prediction for traversal of symmetric IPv4 NATs, and explicit port mapping using uPnP and/or NAT-PMP if these are available on the local physical LAN. -This is not a wholly unique design. It shares features in common with STUN/ICE, SIP, and other protocols. The most novel aspect may be the simplification achieved through lazy transport triggered link provisioning. This trades the [complicated state machines of STUN/ICE](https://www.pkcsecurity.com/untangling-webrtc-flow.html) for a stateless algorithm with implicit empirical parameters. It also eliminates asymmetry. As mentioned above, roots run the same code as regular nodes. +*[A blog post from 2014 by ZeroTier's original author explains some of the reasoning behind VL1's design.](http://adamierymenko.com/decentralization-i-want-to-believe/)* -*[A blog post from 2014 by ZeroTier's original author explains some of the reasoning process that led to this design.](http://adamierymenko.com/decentralization-i-want-to-believe/)* +### **2.1.2.** Addressing -### **2.1.2.** Addressing +Every node is uniquely identified on VL1 by a 40-bit (10 hex digit) **ZeroTier address**. This address is computed from the public portion of a public/private key pair. A node's address, public key, and private key together form its **identity**. -Every device (a "device" can be anything from a laptop to an app) is identified on VL1 by a 40-bit (10 hex digit) unique *ZeroTier address*. These are the addresses used to address packets in the process described in 2.1.1 above. +*On devices running ZeroTier One the node identity is stored in `identity.public` and `identity.secret` in the service's home directory.* -These addresses are computed from the public portion of a public/private key pair. An address along with its public key is called an *identity*. If you look at the home directory of a running ZeroTier instance you will see `identity.public` and `identity.secret`. +When ZeroTier starts for the first time it generates a new identity. It then attempts to advertise it upstream to the network. In the very unlikely event that the identity's 40-bit unique address is taken, it discards it and generates another. -When ZeroTier starts for the first time it generates a new key pair and a new identity. It then attempts to advertise it upstream to the network. In the very unlikely event that the identity's 40-bit unique address is taken, it discards it and generates another. +Identities are claimed on a first come first serve basis and currently expire from planetary roots after 60 days of inactivity. If a long-dormant device returns it may re-claim its identity unless its address has been taken in the meantime (again, highly unlikely). -Identities are claimed on a first come first serve basis and currently expire from global roots after 60 days of inactivity. If a long-dormant device returns it may re-claim its identity unless its address has been taken in the meantime (again, highly unlikely). +The address derivation algorithm used to compute addresses from public keys imposes a computational cost barrier against the intentional generation of a collision. Currently it would take approximately 10,000 CPU-years to do so (assuming e.g. a 3ghz Intel core). This is expensive but not impossible, but it's only the first line of defense. After generating a collision an attacker would then have to compromise all upstream nodes, network controllers, and anything else that has recently communicated with the target node and replace their cached identities. -The address derivation algorithm used to compute addresses from public keys imposes a computational cost barrier against the intentional generation of a collision. Currently it would take approximately 10,000 CPU-years to do so (assuming e.g. a 3ghz Intel core). This is expensive but not impossible, but it's only the first line of defense. After generating a collision an attacker would then have to compromise all upstream nodes and replace the address's cached identity, not to mention also doing the same for peers that have seen the target identity recently. +ZeroTier addresses are, once advertised and claimed, a very secure method of unique identification. -In addition to assisting with communication, upstream nodes also act as identity caches. If the identity corresponding to an address is not known a peer may request it by sending a message called *WHOIS* upstream. +When a node attempts to send a message to another node whose identity is not cached, it sends a *whois* query upstream to a root. Roots provide an authoritative identity cache. -### **2.1.3.** Encryption and Authentication +### **2.1.3.** Cryptography If you don't know much about cryptography you can safely skip this section. **TL;DR: packets are end-to-end encrypted and can't be read by roots or anyone else, and we use modern 256-bit crypto in ways recommended by the professional cryptographers that created it.** Asymmetric public key encryption is [Curve25519/Ed25519](https://en.wikipedia.org/wiki/Curve25519), a 256-bit elliptic curve variant. -Every VL1 packet is encrypted end to end using (as of the current version) 256-bit [Salsa20](https://ianix.com/pub/salsa20-deployment.html) and authenticated using the [Poly1305](https://en.wikipedia.org/wiki/Poly1305) message authentication (MAC) algorithm. - -MAC is computed after encryption [(encrypt-then-MAC)](https://tonyarcieri.com/all-the-crypto-code-youve-ever-written-is-probably-broken) and the cipher/MAC composition used is identical to the [NaCl reference implementation](https://nacl.cr.yp.to). +Every VL1 packet is encrypted end to end using (as of the current version) 256-bit [Salsa20](https://ianix.com/pub/salsa20-deployment.html) and authenticated using the [Poly1305](https://en.wikipedia.org/wiki/Poly1305) message authentication (MAC) algorithm. MAC is computed after encryption [(encrypt-then-MAC)](https://tonyarcieri.com/all-the-crypto-code-youve-ever-written-is-probably-broken) and the cipher/MAC composition used is identical to the [NaCl reference implementation](https://nacl.cr.yp.to). As of today we do not implement [forward secrecy](https://en.wikipedia.org/wiki/Forward_secrecy) or other stateful cryptographic features in VL1. We don't do this for the sake of simplicity, reliability, and code footprint, and because frequently changing state makes features like clustering and fail-over much harder to implement. See [our discussion on GitHub](https://github.com/zerotier/ZeroTierOne/issues/204). -For those who have very high security needs and want forward secrecy, we currently recommend the use of encrypted protocols such as SSH and SSL over ZeroTier. Not only do these provide forward secrecy, the use of multiple layers of encryption in this way provides excellent [defense in depth](https://en.wikipedia.org/wiki/Defense_in_depth_%28computing%29). The computational cost of this additional crypto is typically small, and the benefit can potentially be large. All software can contain bugs, but multiple layers of protection means that discovery of a catastrophic bug in any one layer does not result in compromise of your entire system. We recommend the same for authentication. While ZeroTier VL2 provides certificate-based network boundary enforcement, we do not recommend that users rely solely on this for access control to critical systems. It is always good to use more than one security measure whenever practical. +We may implement forward secrecy in the future. For those who want this level of security today, we recommend using other cryptographic protocols such as SSL or SSH over ZeroTier. These protocols typically implement forward secrecy, but using them over ZeroTier also provides the secondary benefit of defense in depth. Most cryptography is compromised not by a flaw in encryption but through bugs in the implementation. If you're using two secure transports, the odds of a critical bug being discovered in both at the same time is very low. The CPU overhead of double-encryption is not significant for most work loads. -### **2.1.4.** Trusted Paths for Fast Local SDN +### **2.1.4.** Trusted Paths for Fast Local SDN -To support the use of ZeroTier as a high performance SDN/NFV protocol over physically secure networks the protocol supports a feature called *trusted paths*. It is possible to configure all ZeroTier devices on a given network to skip encryption and authentication for traffic over a designated physical path. This can cut CPU use noticably in high traffic scenarios but at the expense of effectively all transport security over the configured trusted backplane. +To support the use of ZeroTier as a high performance SDN/NFV protocol over physically secure networks the protocol supports a feature called *trusted paths*. It is possible to configure all ZeroTier devices on a given network to skip encryption and authentication for traffic over a designated physical path. This can cut CPU use noticably in high traffic scenarios but at the cost of losing virtually all transport security. Trusted paths do not prevent communication with devices elsewhere, since traffic over other paths will be encrypted and authenticated normally. -We don't recommend the use of this feature unless you really need the performance and you know what you're doing. Extra security is never a bad thing. We also recommend thinking carefully before disabling transport security on a cloud private network. Larger cloud providers such as Amazon and Azure tend to provide good network segregation but many less costly providers offer private networks that are "party lines." For these the encryption and authentication provided by ZeroTier is very desirable. In fact, we have a few users using ZeroTier exactly for this reason. +We don't recommend the use of this feature unless you really need the performance and you know what you're doing. We also recommend thinking carefully before disabling transport security on a cloud private network. Larger cloud providers such as Amazon and Azure tend to provide good network segregation but many less costly providers offer private networks that are "party lines" and are not much more secure than the open Internet. + +### **2.2.** VL2: The Ethernet Virtualization Layer + +**VL2** is a [VXLAN](https://en.wikipedia.org/wiki/Virtual_Extensible_LAN)-like network virtualization protocol with SDN management features. It implements secure VLAN boundaries, multicast, rules, capability based security, and certificate based access control. + +VL2 is built atop and carried by VL1, and in so doing it inherits VL1's encryption and endpoint authentication and can use VL1 asymmetric keys to sign and verify credentials. VL1 also allows us to implement VL2 entirely free of concern for underlying physical network topology. Connectivity and routing efficiency issues are VL1 concerns. It's important to understand that there is no relationship between VL2 virtual networks and VL1 paths. Much like VLAN multiplexing on a wired LAN, two nodes that share multiple network memberships in common will still only have one VL1 path (virtual wire) between them. + +### **2.2.1.** Network Identifiers and Controllers + +Each VL2 network (VLAN) is identified by a 64-bit (16 hex digit) **ZeroTier network ID** that contains the 40-bit ZeroTier address of the network's **controller** and a 24-bit number identifying the network on the controller. + + Network ID: 8056c2e21c123456 + | | + | Network number on controller + | + ZeroTier address of controller + +When a node joins a network or requests a network configuration update, it sends a network config query message (via VL1) to the network's controller. The controller can then use the node's VL1 address to look it up on the network and send it the appropriate certificates, credentials, and configuration information. From the perpsective of VL2 virtual networks, VL1 ZeroTier addresses can be thought of as port numbers on an enormous global-scale virtual switch. + +A common misunderstanding is to conflate network controllers with root servers (planet and moons). Root servers are connection facilitators that operate at the VL1 level. Network controllers are configuration managers and certificate authorities that belong to VL2. Generally root servers don't join or control virtual networks and network controllers are not root servers, though it is possible to have a node do both. + +#### Controller Security Considerations + +Network controllers serve as certificate authorities for ZeroTier virtual networks. As such, their `identity.secret` files should be guarded closely and backed up securely. Compromise of a controller's secret key would allow an attacker to issue fraudulent network configurations or admit unauthorized members, while loss of the secret key results in loss of ability to control the network in any way or issue configuration updates and effectively renders the network unusable. + +It is important that controllers' system clocks remain relatively accurate (to within 30-60 seconds) and that they are secure against remote tampering. Many cloud providers provide secure time sources either directly via the hypervisor or via NTP servers within their networks. + +### **2.2.2.** Certificates and Other Credentials + +All credentials issued by network controllers to member nodes in a given network are signed by the controller's secret key to allow all network members to verify them. Credentials have timestamp fields populated by the controller, allowing relative comparison without the need to trust the node's local system clock. + +Credentials are issued only to their owners and are then pushed peer to peer by nodes that wish to communicate with other nodes on the network. This allows networks to grow to enormous sizes without requiring nodes to cache large numbers of credentials or to constantly consult the controller. + +#### Credential Types + + * **Certificates of Membership**: a certificate that a node presents to obtain the right to communicate on a given network. Certificates of membership are accepted if they *agree*, meaning that the submitting member's certificate's timestamp differs from the recipient's certificate's timestamp by no more than the recipient certificate's maximum timestamp delta value. This creates a decentralized moving-window scheme for certificate expiration without requiring node clock synchronization or constant checking with the controller. + + * **Revocations**: a revocation instantaneously revokes a given credential by setting a hard timestamp limit before which it will not be accepted. Revocations are rapidly propagated peer to peer among members of a network using a rumor mill algorithm, allowing a controller to revoke a member credential across the entire network even if its connection to some members is unreliable. + + * **Capabilities**: a capability is a bundle of network rules that is signed by the controller and can be presented to other members of a network to grant the presenter elevated privileges within the framework of the network's base rule set. More on this in the section on rules. + + * **Tags**: a tag is a key/value pair signed by the controller that is automatically presented by members to one another and can be matched on in base or capability network rules. Tags can be used to categorize members by role, department, classification, etc. + + * **Certificates of Ownership**: these certify that a given network member owns something, such as an IP address. These are currently only used to lock down networks against IP address spoofing but could be used in the future to certify ownership of other network-level entities that can be matched in a filter. + +### **2.2.3.** Multicast, ARP, NDP, and Special Addressing Modes + +ZeroTier networks support multicast via a simple publish/subscribe system. + +When a node wishes to receive multicasts for a given multicast group, it advertises membership in this group to other members of the network with which it is communicating and to the network controller. When a node wishes to send a multicast it both consults its cache of recent advertisements and periodically solicits additional advertisements. + +Broadcast (Ethernet *ff:ff:ff:ff:ff:ff*) is treated as a multicast group to which all members subscribe. It can be disabled at the network level to reduce traffic if it is not needed. IPv4 ARP receives special handling (see below) and will still work if normal broadcast is disabled. + +Multicasts are propagated using simple sender-side replication. This places the full outbound bandwidth load for multicast on the sender and minimizes multicast latency. Network configurations contain a network-wide **multicast limit** configurable at the network controller. This specifies the maximum number of other nodes to which any node will send a multicast. If the number of known recipients in a given multicast group exceeds the multicast limit, the sender chooses a random subset. + +There is no global limit on multicast recipients, but setting the multicast limit very high on very large networks could result in significant bandwidth overhead. + +#### Special Handling of IPv4 ARP Broadcasts + +IPv4 [ARP](https://en.wikipedia.org/wiki/Address_Resolution_Protocol) is built on simple Ethernet broadcast and scales poorly on large or distributed networks. To improve ARP's scalability ZeroTier generates a unique multicast group for each IPv4 address detected on its system and then transparently intercepts ARP queries and sends them only to the correct group. This converts ARP into effectively a unicast or narrow multicast protocol (like IPv6 NDP) and allows IPv4 ARP to work reliably across wide area networks without excess bandwidth consumption. A similar strategy is implemented under the hood by a number of enterprise switches and WiFi routers designed for deployment on extremely large LANs. This ARP emulation mode is transparent to the OS and application layers, but it does mean that packet sniffers will not see all ARP queries on a virtual network the way they typically can on smaller wired LANs. + +#### Multicast-Free IPv6 Addressing Modes + +IPv6 uses a protocol called [NDP](https://en.wikipedia.org/wiki/Neighbor_Discovery_Protocol) in place of ARP. It is similar in role and design but uses narrow multicast in place of broadcast for superior scalability on large networks. This protocol nevertheless still imposes the latency of an additional multicast lookup whenever a new address is contacted. This can add hundreds of milliseconds over a wide area network, or more if latencies associated with pub/sub recipient lookup are significant. + +IPv6 addresses are large enough to easily encode ZeroTier addresses. For faster operation and better scaling we've implemented several special IPv6 addressing modes that allow the local node to emulate NDP. These are ZeroTier's **rfc4193** and **6plane** IPv6 address assignment schemes. If these addressing schemes are enabled on a network, nodes locally intercept outbound NDP queries for matching addresses and then locally generate spoofed NDP replies. + +Both modes dramatically reduce initial connection latency between network members. **6plane** additionally exploits NDP emulation to transparently assign an entire IPv6 /80 prefix to every node without requiring any node to possess additional routing table entries. This is designed for virtual machine and container hosts that wish to auto-assign IPv6 addresses to guests and is very useful on microservice architecture backplane networks. + +Finally there is a security benefit to NDP emulation. ZeroTier addresses are cryptographically authenticated, and since Ethernet MAC addresses on networks are computed from ZeroTier addresses these are also secure. NDP emulated IPv6 addressing modes are therefore not vulnerable to NDP reply spoofing. + +Normal non-NDP-emulated IPv6 addresses (including link-local addresses) can coexist with NDP-emulated addressing schemes. Any NDP queries that do not match NDP-emulated addresses are sent via normal multicast. + +### **2.2.4.** Ethernet Bridging + +ZeroTier emulates a true Ethernet switch. This includes the ability to L2 bridge other Ethernet networks (wired LAN, WiFi, virtual backplanes, etc.) to virtual networks using conventional Ethernet bridging. + +To act as a bridge a network member must be designated as such by the controller. This is for security reasons as normal network members are not permitted to send traffic from any origin other than their MAC address. Designated bridges also receive special treatment from the multicast algorithm, which more aggressively and directly queries them for group subscriptions and replicates all broadcast traffic and ARP requests to them. As a result bridge nodes experience a slightly higher amount of multicast bandwidth overhead. + +Bridging has been tested extensively on Linux using the Linux kernel native bridge, which cleanly handles network MTU mismatch. There are third party reports of bridging working on other platforms. The details of setting up bridging, including how to selectively block traffic like DHCP that may not be wanted across the bridge, are beyond the scope of this manual. + +### **2.2.5.** Public Networks + +It is possible to disable access control on a ZeroTier network. A public network's members do not check certificates of membership, and new members to a public network are automatically marked as authorized by their host controller. It is not possible to de-authorize a member from a public network. + +Rules on the other hand *are* enforced, so it's possible to implement a special purpose public network that only allows access to a few things or that only allows a restricted subset of traffic. + +Public networks are useful for testing and for peer to peer "party lines" for gaming, chat, and other applications. Participants in public networks are warned to pay special attention to security. If joining a public network be careful not to expose vulnerable services or accidentally share private files via open network shares or HTTP servers. Make sure your operating system, applications, and services are fully up to date. + +ZeroTier, Inc. operates a public network called Earth (no relation to the root server planet definition of the same name) with the network ID `8056c2e21c000001`. Earth issues IPv4 addresses in the unused IPv4 space 28.0.0.0/7 and rfc4193 IPv6 addresses and allows multicast for service discovery. It's essentially a global LAN party. After joining Earth visit `http://earth.zerotier.net/` to get a page showing your Earth virtual IP address and Ethernet MAC address. + +### **2.2.6.** Ad-Hoc Networks + +A special kind of public network called an ad-hoc network may be accessed by joining a network ID with the format: + + ffSSSSEEEE000000 + | | | | + | | | Reserved for future use, must be 0 + | | End of port range (hex) + | Start of port range (hex) + Reserved ZeroTier address prefix indicating a controller-less network + +Ad-hoc networks are public (no access control) networks that have no network controller. Instead their configuration and other credentials are generated locally. Ad-hoc networks permit only IPv6 UDP and TCP unicast traffic (no multicast or broadcast) using 6plane format NDP-emulated IPv6 addresses. In addition an ad-hoc network ID encodes an IP port range. UDP packets and TCP SYN (connection open) packets are only allowed to desintation ports within the encoded range. + +For example `ff00160016000000` is an ad-hoc network allowing only SSH, while `ff0000ffff000000` is an ad-hoc network allowing any UDP or TCP port. + +Keep in mind that these networks are public and anyone in the entire world can join them. Care must be taken to avoid exposing vulnerable services or sharing unwanted files or other resources. + +## **3.** The Network Rules Engine + +Traffic on ZeroTier networks can be observed and controlled with a system of globally applied network rules. These are enforced in a distributed fashion by both the senders and the receivers of packets, meaning that to escape the rules engine a malicious attacker would need to compromise systems on both sides of the conversation. + +The ZeroTier VL2 rules engine differs from most other firewalls and SDN rules engines in several ways. The most immediately relevant of these is that the ZeroTier rules engine is stateless, meaning it lacks connection tracking. This means that bidirectional whitelisting can't be accomplished by simply whitelisting reply packets to established connections. Instead some thought must be put into how to allow both sides of a desired flow. Rule patterns to achieve the most common desired objectives are included in this manual. + +The decision to make our rules engine stateless was a design trade-off driven by several concerns. First we wanted to keep complexity, code footprint, and memory use very low to support small embedded devices. The second and more fundamental reason is that distributed stateful filtering requires distributed state synchronization. This would have added a large volume of additional sync traffic as well as introducing [inescapable](https://en.wikipedia.org/wiki/CAP_theorem) new sources of instablity and failure and a lot of surface area for security vulnerabilities. + +While ZeroTier lacks state tracking, its rules engine includes something not found anywhere else in the enterprise networking space: [capability-based security](https://en.wikipedia.org/wiki/Capability-based_security) and device tagging. Capabilities and tags allow extremely complex micro-segmented network rule schemes to be implemented in a sane, conceptual way that is both easier for human beings to understand and more efficient for machines to handle. + +This section assumes some level of familiarity with network rules as they're commonly used on firewalls and routers, etc. While the rules engine is part of VL2, it's been given its own section in this manual due to the depth and cross-cutting nature of the topic. + +### **3.1.** Rule Sets and Rule Evaluation + +Rule sets are ordered lists of one or more rules, with each rule consisting of one or more **match** conditions followed by one **action**. As a rule set is evaluated, each match is tested in order and is then ANDed or ORed with the previous match result state. When an action is encountered it is taken if the result of the preceding matches is *true*. An action with no preceding matches is always taken. If no permissive actions are taken by any rule set the packet is discarded. + +Here is a simple rule set that constrains Ethernet traffic on a network to only IPv4, ARP, or IPv6 as it would appear in the raw JSON format used by ZeroTier One's built-in network controller implementation. Don't worry if this seems verbose and difficult. We have a more human-friendly way of writing rule sets, but before we introduce it it's important to understand what is really happening. + + [ + { + "etherType": 2048, + "not": true, + "or": false, + "type": "MATCH_ETHERTYPE" + }, + { + "etherType": 2054, + "not": true, + "or": false, + "type": "MATCH_ETHERTYPE" + }, + { + "etherType": 34525, + "not": true, + "or": false, + "type": "MATCH_ETHERTYPE" + }, + { + "type": "ACTION_DROP" + }, + { + "type": "ACTION_ACCEPT" + } + ] + +This checks whether an Ethernet level packet is _not_ IPv4 (ethertype 2048) _and not_ IPv4 ARP (ethertype 2054) _and not_ IPv6 (ethertype 34525). If all three matches evaluate to true (meaning the ethertype is none of these) then the **drop** action is taken. Otherwise the **accept** action is taken. + +Networks have one base rule set that is applied to all traffic. Its size is constrained to 1024 entries (each match or action is an entry). It should be used to set the overall policies for all members of the network, and for most common use cases it's all you'll need. For more complex scenarios, both capabilities and tags provide methods of both managing complexity and scaling the overall size of a network's rule system. + +### **3.2.** Capabilities + +A capability is a small rule set that is bundled into a credential object, signed by the network controller, and issued to only those member(s) permitted to exercise it. When a member detects that outgoing traffic does not match the base rule set but is allowed by one of its capabilities, it periodically pushes the matching capability credential to the recipient ahead of the packet(s) in question. Peer to peer capability distribution is automatic and is triggered by capability match. + +When the recipient receives the capability it authenticates it by checking its signature and timestamp and, provided the capability is valid, adds it to the set of capabilities to apply to incoming traffic from the capability's owner. The sender has effectively told the recipient "I can too send this packet! Teacher says so!" + +Capabilities allow large systems of rules to be broken down into functional aspects and then distributed intelligently only to those members with a need to know. This avoids the bandwidth and storage overhead of distributing huge monolithic rule sets and organizes rules conceptually to make them easier for administrators to understand. + +There are three terminating actions that can be taken in a rule set: **accept**, **break**, and **drop**. The accept action terminates rule evaluation and accepts the packet. The break action terminates the evaluation of the current rule set but permits the further evaluation of capabilities. The drop action terminates rule evaluation and drops the packet without checking capabilities in the base rule set, but is equivalent to break in capability rule sets. In most cases break should be used unless certain traffic must be absolutely prohibited under any circumstance. + +In the simple base rule set example in section 3.1 the drop action is taken in the unapproved case. This means that ethernet whitelisting cannot be overridden by a capability. If we change `ACTION_DROP` in our example to `ACTION_BREAK`, then it becomes possible to issue the following capability: + + [ + { + "etherType": 2114, + "not": false, + "or": false, + "type": "MATCH_ETHERTYPE" + }, + { + "type": "ACTION_ACCEPT" + } + ] + +Ethertype 2114 is [wake-on-LAN](https://wiki.wireshark.org/WakeOnLAN), a special packet that can cause some systems to wake from sleep mode. If we place the above tiny rule set into a capability and issue it to a device, this device *but no others* will now be permitted to send wake-on-LAN magic packets. (Wake-on-LAN requires hardware support so it would only work to target devices plugged into a physical network bridged to a ZeroTier network, but don't worry about that here. It's just an example of special traffic.) + +Capability rule sets are limited to only 64 entries. The idea is to keep them small and simple. A capability should grant one thing or one small set of conceptually related things. + +### **3.3.** Tags + +ZeroTier provides a second mechanism to control rule set complexity. Tags are 32-bit numeric key-value pair credentials that are issued to network members and signed by the controller. They are then distributed peer to peer on a need to know basis in a similar manner to capabilities. + +Tags provide a way to conditionally drop or allow traffic between members by member classification. They allow very detailed network micro-segmentation by member role, permission, function, etc. without resulting in a combinatorial explosion in rules table size. + +Let's say we want to permit traffic on TCP ports 139 and 445 (netbios/CIFS file sharing) only between systems that belong to the same department. Our company has 12,000 devices and 10 departments. Without tags this would require 144,000,000 rules, but with tags it can be accomplished by only a few. + +First a tag is created to represent the department. Let's give it tag ID 100. Each member system receives the tag with a value from 1 to 10 indicating which department it belongs to. We can then add the following rules to our network's base rule set (or to a capability if so desired): -### **2.1.5.** Troubleshooting Connectivity Problems + [ + { + "type": "MATCH_IP_DEST_PORT_RANGE", + "not": false, + "or": false, + "start": 139, + "end": 139 + }, + { + "type": "MATCH_IP_DEST_PORT_RANGE", + "not": false, + "or": true, + "start": 445, + "end": 445 + }, + { + "type": "MATCH_IP_PROTOCOL", + "not": false, + "or": false, + "ipProtocol": 6 + }, + { + "type": "MATCH_TAGS_DIFFERENCE", + "not": false, + "or": false, + "id": 10, + "value": 0 + }, + { + "type": "ACTION_ACCEPT" + } + ] + +This tells members in our network to accept TCP packets on ports 139 or 445 if the difference between tags with tag ID 10 is zero, meaning they match. (If a member does not have a value for this tag, it does not match.) Now all members of the same department can access CIFS file shares, but CIFS sharing between departments could still be prohibited. (TCP whitelisting requires some additional rules due to the stateless nature of our rules engine. See the section below on rule design patterns.) + +Tags can be compared on numeric value or as bit fields via several different bit mask operations allowing many different systems of member classification to be implemented. + +### **3.4.** Rule Description Language -- cgit v1.2.3 From d09d193715646ecad050d7f6d3d23ae7963831d6 Mon Sep 17 00:00:00 2001 From: Adam Ierymenko Date: Mon, 13 Mar 2017 09:37:24 -0700 Subject: release notes, and delete MANUAL from this repo for now since it isn't quite done and will take shape on the web site --- RELEASE-NOTES.md | 92 +++++++++++++++ doc/MANUAL.md | 354 ------------------------------------------------------- 2 files changed, 92 insertions(+), 354 deletions(-) create mode 100644 RELEASE-NOTES.md delete mode 100644 doc/MANUAL.md (limited to 'doc') diff --git a/RELEASE-NOTES.md b/RELEASE-NOTES.md new file mode 100644 index 00000000..42a2aaa4 --- /dev/null +++ b/RELEASE-NOTES.md @@ -0,0 +1,92 @@ +ZeroTier Release Notes +====== + +*As of 1.2.0 this will serve as a detailed changelog, which we've needed for a long time.* + +# 2017-03-13 -- Version 1.2.0 + +Version 1.2.0 is a major milestone release and introduces a large number of new capabilities to the ZeroTier core network hypervisor. It also includes some security tightening, major UI improvements for Windows and Macintosh platforms, and a number of bug fixes and platform issue workarounds. + +## Features in 1.2.0 + +### The ZeroTier Rules Engine + +The largest new feature in 1.2.0, and the product of many months of work, is our advanced network rules engine. With this release we achieve traffic control, security monitoring, and micro-segmentation capability on par with many enterprise SDN solutions designed for use in advanced data centers and corporate networks. + +Rules allow you to filter packets on your network and vector traffic to security observers (e.g. a node running Snort). Security observation can be performed in-band using REDIRECT or out of band using TEE, and for tha latter it can be done for headers only, for select traffic, or probabilistically to reduce overhead on large distributed networks. + +Tags and capabilites provide advanced methods for implementing fine grained permission structures and micro-segmentation schemes without bloating the size and complexity of your rules table. + +See our manual for more information. + +### Root Server Federation + +It's now possible to create your own root servers and add them to the root server pool on your nodes. This is done by creating what's called a "moon," which is a signed enumeration of root servers and their stable points on the network. Refer to the manual for more details on how to do this and how it works. + +Federated roots achieve a number of things: + + * You can deploy your own infrastructure to reduce dependency on ours. + * You can deploy them *inside your LAN* to ensure that network connectivity inside your facility still works if the Internet goes down. This is the first step toward making ZeroTier viable as an in-house SDN solution. + * Roots can be deployed inside national boundaries for countries with data residency laws or "great firewalls." (As of 1.2.0 there is still no way to force all traffic to use these roots, but that will be easy to do in a later version.) + * Last but not least this makes ZeroTier somewhat less centralized by eliminating any hard dependency on ZeroTier, Inc.'s infrastructure. + +Our roots will of course remain and continue to provide zero-configuration instant-on deployment, a secure global authority for identities, and free traffic relaying for those who can't establish peer to peer connections. + +### Local Configuration + +An element of our design philosophy is "features are bugs." This isn't an absolute dogma but more of a guiding principle. We try as hard as we can to avoid adding features, especially "knobs" that must be tweaked by a user. + +As of 1.2.0 we've decided that certain knobs are unavoidable, and so there is now a `local.conf` file that can be used to configure them. See the ZeroTier One documentation for these. They include: + + * Blacklisting interfaces you want to make sure ZeroTier doesn't use for network traffic, such as VPNs, slow links, or backplanes designated for only certain kinds of traffic. + * Turning uPnP/NAT-PMP on or off. + * Configuring software updates on Windows and Mac platforms. + * Defining trusted paths (the old trusted paths file is now deprecated) + * Setting the ZeroTier main port so it doesn't have to be changed on the command line, which is very inconvenient in many cases. + +### Improved In-Band Software Updates + +A good software update system for Windows and Mac clients has been a missing feature in previous versions. It does exist but we've been shy about using it so far due to its fragility in some environments. + +We've greatly improved this mechanism in 1.2.0. Not only does it now do a better job of actually invoking the update, but it also transfers updates in-band using the ZeroTier protocol. This means it can work in environments that do not allows http/https traffic or that force it through proxies. There's also now an update channel setting: `beta` or `release` (the default). + +As before software updates are authenticated in two ways: + + 1. ZeroTier's own signing key is used to sign all updates and this signature is checked prior to installation. Our signatures are done on an air-gapped machine. + + 2. Updates for Mac and Windows are signed using Apple and Microsoft (DigiCert) keys and will not install unless these signatures are also valid. + +Version 1.2.0's in-band mechanism effectively adds a third way: updates are fetched in-band from a designated ZeroTier node, thus authenticating the source using ZeroTier's built-in encryption and authentication mechanisms. + +Updates are now configurable via `local.conf`. There are three options: `disable`, `download`, and `apply`. The third is the default for official builds on Windows and Mac, making updates happen silently and automatically as they do for popular browsers like Chrome and Firefox. For managed enterprise deployments IT people could ship a local.conf that disables updates and instead push updates via their management capabilities. Updates are disabled on Linux and other Unix-type platforms as these get updates through package repositories. + +### Path Quality Monitoring (QoS and SD-WAN phase one) + +Version 1.2.0 is now aware of the link quality of direct paths with other 1.2.0 nodes. This information isn't used yet but is visible through the JSON API. (Quality always shows as 100% with pre-1.2.0 nodes.) + +Link quality monitoring is a precursor to intelligent multi-path and QoS support, which will in future versions bring us to feature parity with SD-WAN products like Cisco iWAN. + +"Connect all the things!" + +### Security Improvements + +Version 1.2.0 adds anti-DOS (denial of service) rate limits and other hardening for improved resiliency against a number of denial of service attack scenarios. + +It also adds a mechanism for instantaneous credential revocation. This can be used to revoke certificates of membership instantly to kick a node off a network (for private networks) and also to revoke capabilities and tags. The new controller sends revocations by default when a peer is de-authorized. + +Revocations propagate using a "rumor mill" peer to peer algorithm. This means that a controller need only successfully send a revocation to at least one member of a network with connections to other active members. At this point the revocation will flood through the network peer to peer very quickly. This helps make revocations more robust in the face of poor connectivity with the controller or attempts to incapacitate the controller with denial of service attacks, as well as making revocations faster on huge networks. + +### Windows and Macintosh UI Improvements (ZeroTier One) + +The Mac has a whole new UI built natively in Objective-C. It provides a pulldown similar in appearance and operation to the Mac WiFi task bar menu. The Windows UI has also been improved and now provides a task bar icon that can be right-clicked to manage networks. Both now expose managed route and IP permissions, allowing nodes to easily opt in to full tunnel operation if you have a router configured on your network. + +## Major Bug Fixes in 1.2.0 + + * **The Windows HyperV 100% CPU bug** + * This long-running problem turns out to have been an issue with Windows itself, but one we were triggering by placing invalid data into the Windows registry. Microsoft is aware of the issue but we've also fixed the triggering problem on our side. ZeroTier should now co-exist quite well with HyperV and should now be able to be bridged with a HyperV virtual switch. + * **Segmenation Faults on musl-libc based Linux Systems** + * Alpine Linux and some embedded Linux systems that use musl libc (a minimal libc) experienced segmentation faults. These were due to a smaller default stack size. A work-around that sets the stack size for new threads has been added. + * **Windows Firewall Blocks Local JSON API** + * On some Windows systems the firewall likes to block 127.0.0.1:9993 for mysterious reasons. This is now fixed in the installer via the addition of another firewall exemption rule. + * **UI Crash on Embedded Windows Due to Missing Fonts** + * The MSI installer now ships fonts and will install them if they are not present, so this should be fixed. diff --git a/doc/MANUAL.md b/doc/MANUAL.md deleted file mode 100644 index b26ad3a5..00000000 --- a/doc/MANUAL.md +++ /dev/null @@ -1,354 +0,0 @@ -ZeroTier Manual -====== - -## **1.** Introduction - -ZeroTier is a smart Ethernet switch for planet Earth. - -It's a distributed network hypervisor built atop a cryptographically secure global peer to peer network that provides capabilities similar to an enterprise smart switch such as VLANs, rules, device authentication, and security monitoring. - -This manual describes the design and operation of ZeroTier and its associated services, apps, and libraries. Its intended audience includes IT professionals, network administrators, information security experts, and developers. - -The first section (2) of this guide explains ZeroTier's design and operation at a high level and is written for those with at least an intermediate knowledge of topics like TCP/IP and Ethernet networking. It's not required reading for most users, but understanding how things work in detail helps clarify everything else and helps tremendously with troubleshooting should anything go wrong. - -The remaining sections deal more concretely with deployment and administration. - -### Table of Contents - -1. [Introduction](#1) -2. [Network Hypervisor Overview](#2) - 1. [VL1: The ZeroTier Peer to Peer Network](#2_1) - 1. [Network Topology and Peer Discovery](#2_1_1) - 2. [Addressing](#2_1_2) - 3. [Cryptography](#2_1_3) - 4. [Trusted Paths for Fast Local SDN](#2_1_4) - 2. [VL2: The Ethernet Virtualization Layer](#2_2) - 1. [Network Identifiers and Controllers](#2_2_1) - 2. [Certificates and Other Credentials](#2_2_2) - 3. [Multicast, ARP, NDP, and Special Addressing Modes](#2_2_3) - 4. [Ethernet Bridging](#2_2_4) - 5. [Public Networks](#2_2_5) - 6. [Ad-Hoc Networks](#2_2_6) -3. [The Network Rules Engine](#3) - 1. [Rule Sets and Rule Evaluation](#3_1) - 2. [Capabilities](#3_2) - 3. [Tags](#3_3) - 4. [Rule Description Language](#3_4) - 1. [Syntax](#3_4_1) - 2. [Actions](#3_4_2) - 3. [Match Conditions](#3_4_3) - 4. [Capabilities](#3_4_4) - 5. [Tags](#3_4_5) - 6. [Macros](#3_4_6) - 5. [Design Patterns](#3_5) - 1. [TCP Whitelisting](#3_5_1) - 2. [Low-Overhead Network Monitoring](#3_5_2) - ------- - -## **2.** Network Hypervisor Overview - -The ZeroTier network hypervisor (currently found in the [node/](https://github.com/zerotier/ZeroTierOne/tree/master/node) subfolder of the ZeroTierOne git repository) is a self-contained network virtualization engine that implements an Ethernet virtualization layer similar to [VXLAN](https://en.wikipedia.org/wiki/Virtual_Extensible_LAN) on top of a global encrypted peer to peer network. - -The ZeroTier protocol is original, though aspects of it are similar to VXLAN and IPSec. It has two conceptually separate but closely coupled layers [in the OSI model](https://en.wikipedia.org/wiki/OSI_model) sense: **VL1** and **VL2**. VL1 is the underlying peer to peer transport layer, the "virtual wire," while VL2 is an emulated Ethernet layer that provides operating systems and apps with a familiar communication medium. - -### **2.1.** VL1: The ZeroTier Peer to Peer Network - -A global data center requires a global wire closet. - -In conventional networks L1 (OSI layer 1) refers to the actual CAT5/CAT6 cables or wireless radio channels over which data is carried and the physical transciever chips that modulate and demodulate it. VL1 is a peer to peer network that does the same thing by using encryption, authentication, and a lot of networking tricks to create virtual wires on a dyniamic as-needed basis. - -### **2.1.1.** Network Topology and Peer Discovery - -VL1 is designed to be zero-configuration. A user can start a new ZeroTier node without having to write configuration files or provide the IP addresses of other nodes. It's also designed to be fast. Any two devices in the world should be able to locate each other and communicate almost instantly. - -To achieve this VL1 is organized like DNS. At the base of the network is a collection of always-present **root servers** whose role is similar to that of [DNS root name servers](https://en.wikipedia.org/wiki/Root_name_server). Roots run the same software as regular endpoints but reside at fast stable locations on the network and are designated as such by a **world definition**. World definitions come in two forms: the **planet** and one or more **moons**. The protocol includes a secure mechanism allowing world definitions to be updated in-band if root servers' IP addresses or ZeroTier addresses change. - -There is only one planet. Earth's root servers are operated by ZeroTier, Inc. as a free service. There are currently twelve root servers organized into two six-member clusters distributed across every major continent and multiple network providers. Almost everyone in the world has one within less than 100ms network latency from their location. - -A node can "orbit" any number of moons. A moon is just a convenient way to add user-defined root servers to the pool. Users can create moons to reduce dependency on ZeroTier, Inc. infrastructure or to locate root servers closer for better performance. For on-premise SDN use a cluster of root servers can be located inside a building or data center so that ZeroTier can continue to operate normally if Internet connectivity is lost. - -Nodes start with no direct links to one another, only upstream to roots (planet and moons). Every peer on VL1 possesses a globally unique 40-bit (10 hex digit) **ZeroTier address**, but unlike IP addresses these are opaque cryptographic identifiers that encode no routing information. To communicate peers first send packets "up" the tree, and as these packets traverse the network they trigger the opportunistic creation of direct links along the way. The tree is constantly trying to "collapse itself" to optimize itself to the pattern of traffic it is carrying. - -Peer to peer connection setup goes like this: - -1. A wants to send a packet to B, but since it has no direct path it sends it upstream to R (a root). -2. If R has a direct link to B, it forwards the packet there. Otherwise it sends the packet upstream until planetary roots are reached. Planetary roots know about all nodes, so eventually the packet will reach B if B is online. -3. R also sends a message called *rendezvous* to A containing hints about how it might reach B. Meanwhile the root that forwards the packet to B sends *rendezvous* informing B how it might reach A. -4. A and B get their *rendezvous* messages and attempt to send test messages to each other, possibly accomplishing [hole punching](https://en.wikipedia.org/wiki/UDP_hole_punching) of any NATs or stateful firewalls that happen to be in the way. If this works a direct link is established and packets no longer need to take the scenic route. - -Since roots forward packets, A and B can reach each other instantly. A and B then begin attempting to make a direct peer to peer connection. If this succeeds it results in a faster lower latency link. We call this *transport triggered link provisioning* since it's the forwarding of the packet itself that triggers the peer to peer network to attempt direct connection. - -VL1 never gives up. If a direct path can't be established, communication can continue through (slower) relaying. Direct connection attempts continue forever on a periodic basis. VL1 also has other features for establishing direct connectivity including LAN peer discovery, port prediction for traversal of symmetric IPv4 NATs, and explicit port mapping using uPnP and/or NAT-PMP if these are available on the local physical LAN. - -*[A blog post from 2014 by ZeroTier's original author explains some of the reasoning behind VL1's design.](http://adamierymenko.com/decentralization-i-want-to-believe/)* - -### **2.1.2.** Addressing - -Every node is uniquely identified on VL1 by a 40-bit (10 hex digit) **ZeroTier address**. This address is computed from the public portion of a public/private key pair. A node's address, public key, and private key together form its **identity**. - -*On devices running ZeroTier One the node identity is stored in `identity.public` and `identity.secret` in the service's home directory.* - -When ZeroTier starts for the first time it generates a new identity. It then attempts to advertise it upstream to the network. In the very unlikely event that the identity's 40-bit unique address is taken, it discards it and generates another. - -Identities are claimed on a first come first serve basis and currently expire from planetary roots after 60 days of inactivity. If a long-dormant device returns it may re-claim its identity unless its address has been taken in the meantime (again, highly unlikely). - -The address derivation algorithm used to compute addresses from public keys imposes a computational cost barrier against the intentional generation of a collision. Currently it would take approximately 10,000 CPU-years to do so (assuming e.g. a 3ghz Intel core). This is expensive but not impossible, but it's only the first line of defense. After generating a collision an attacker would then have to compromise all upstream nodes, network controllers, and anything else that has recently communicated with the target node and replace their cached identities. - -ZeroTier addresses are, once advertised and claimed, a very secure method of unique identification. - -When a node attempts to send a message to another node whose identity is not cached, it sends a *whois* query upstream to a root. Roots provide an authoritative identity cache. - -### **2.1.3.** Cryptography - -If you don't know much about cryptography you can safely skip this section. **TL;DR: packets are end-to-end encrypted and can't be read by roots or anyone else, and we use modern 256-bit crypto in ways recommended by the professional cryptographers that created it.** - -Asymmetric public key encryption is [Curve25519/Ed25519](https://en.wikipedia.org/wiki/Curve25519), a 256-bit elliptic curve variant. - -Every VL1 packet is encrypted end to end using (as of the current version) 256-bit [Salsa20](https://ianix.com/pub/salsa20-deployment.html) and authenticated using the [Poly1305](https://en.wikipedia.org/wiki/Poly1305) message authentication (MAC) algorithm. MAC is computed after encryption [(encrypt-then-MAC)](https://tonyarcieri.com/all-the-crypto-code-youve-ever-written-is-probably-broken) and the cipher/MAC composition used is identical to the [NaCl reference implementation](https://nacl.cr.yp.to). - -As of today we do not implement [forward secrecy](https://en.wikipedia.org/wiki/Forward_secrecy) or other stateful cryptographic features in VL1. We don't do this for the sake of simplicity, reliability, and code footprint, and because frequently changing state makes features like clustering and fail-over much harder to implement. See [our discussion on GitHub](https://github.com/zerotier/ZeroTierOne/issues/204). - -We may implement forward secrecy in the future. For those who want this level of security today, we recommend using other cryptographic protocols such as SSL or SSH over ZeroTier. These protocols typically implement forward secrecy, but using them over ZeroTier also provides the secondary benefit of defense in depth. Most cryptography is compromised not by a flaw in encryption but through bugs in the implementation. If you're using two secure transports, the odds of a critical bug being discovered in both at the same time is very low. The CPU overhead of double-encryption is not significant for most work loads. - -### **2.1.4.** Trusted Paths for Fast Local SDN - -To support the use of ZeroTier as a high performance SDN/NFV protocol over physically secure networks the protocol supports a feature called *trusted paths*. It is possible to configure all ZeroTier devices on a given network to skip encryption and authentication for traffic over a designated physical path. This can cut CPU use noticably in high traffic scenarios but at the cost of losing virtually all transport security. - -Trusted paths do not prevent communication with devices elsewhere, since traffic over other paths will be encrypted and authenticated normally. - -We don't recommend the use of this feature unless you really need the performance and you know what you're doing. We also recommend thinking carefully before disabling transport security on a cloud private network. Larger cloud providers such as Amazon and Azure tend to provide good network segregation but many less costly providers offer private networks that are "party lines" and are not much more secure than the open Internet. - -### **2.2.** VL2: The Ethernet Virtualization Layer - -**VL2** is a [VXLAN](https://en.wikipedia.org/wiki/Virtual_Extensible_LAN)-like network virtualization protocol with SDN management features. It implements secure VLAN boundaries, multicast, rules, capability based security, and certificate based access control. - -VL2 is built atop and carried by VL1, and in so doing it inherits VL1's encryption and endpoint authentication and can use VL1 asymmetric keys to sign and verify credentials. VL1 also allows us to implement VL2 entirely free of concern for underlying physical network topology. Connectivity and routing efficiency issues are VL1 concerns. It's important to understand that there is no relationship between VL2 virtual networks and VL1 paths. Much like VLAN multiplexing on a wired LAN, two nodes that share multiple network memberships in common will still only have one VL1 path (virtual wire) between them. - -### **2.2.1.** Network Identifiers and Controllers - -Each VL2 network (VLAN) is identified by a 64-bit (16 hex digit) **ZeroTier network ID** that contains the 40-bit ZeroTier address of the network's **controller** and a 24-bit number identifying the network on the controller. - - Network ID: 8056c2e21c123456 - | | - | Network number on controller - | - ZeroTier address of controller - -When a node joins a network or requests a network configuration update, it sends a network config query message (via VL1) to the network's controller. The controller can then use the node's VL1 address to look it up on the network and send it the appropriate certificates, credentials, and configuration information. From the perpsective of VL2 virtual networks, VL1 ZeroTier addresses can be thought of as port numbers on an enormous global-scale virtual switch. - -A common misunderstanding is to conflate network controllers with root servers (planet and moons). Root servers are connection facilitators that operate at the VL1 level. Network controllers are configuration managers and certificate authorities that belong to VL2. Generally root servers don't join or control virtual networks and network controllers are not root servers, though it is possible to have a node do both. - -#### Controller Security Considerations - -Network controllers serve as certificate authorities for ZeroTier virtual networks. As such, their `identity.secret` files should be guarded closely and backed up securely. Compromise of a controller's secret key would allow an attacker to issue fraudulent network configurations or admit unauthorized members, while loss of the secret key results in loss of ability to control the network in any way or issue configuration updates and effectively renders the network unusable. - -It is important that controllers' system clocks remain relatively accurate (to within 30-60 seconds) and that they are secure against remote tampering. Many cloud providers provide secure time sources either directly via the hypervisor or via NTP servers within their networks. - -### **2.2.2.** Certificates and Other Credentials - -All credentials issued by network controllers to member nodes in a given network are signed by the controller's secret key to allow all network members to verify them. Credentials have timestamp fields populated by the controller, allowing relative comparison without the need to trust the node's local system clock. - -Credentials are issued only to their owners and are then pushed peer to peer by nodes that wish to communicate with other nodes on the network. This allows networks to grow to enormous sizes without requiring nodes to cache large numbers of credentials or to constantly consult the controller. - -#### Credential Types - - * **Certificates of Membership**: a certificate that a node presents to obtain the right to communicate on a given network. Certificates of membership are accepted if they *agree*, meaning that the submitting member's certificate's timestamp differs from the recipient's certificate's timestamp by no more than the recipient certificate's maximum timestamp delta value. This creates a decentralized moving-window scheme for certificate expiration without requiring node clock synchronization or constant checking with the controller. - - * **Revocations**: a revocation instantaneously revokes a given credential by setting a hard timestamp limit before which it will not be accepted. Revocations are rapidly propagated peer to peer among members of a network using a rumor mill algorithm, allowing a controller to revoke a member credential across the entire network even if its connection to some members is unreliable. - - * **Capabilities**: a capability is a bundle of network rules that is signed by the controller and can be presented to other members of a network to grant the presenter elevated privileges within the framework of the network's base rule set. More on this in the section on rules. - - * **Tags**: a tag is a key/value pair signed by the controller that is automatically presented by members to one another and can be matched on in base or capability network rules. Tags can be used to categorize members by role, department, classification, etc. - - * **Certificates of Ownership**: these certify that a given network member owns something, such as an IP address. These are currently only used to lock down networks against IP address spoofing but could be used in the future to certify ownership of other network-level entities that can be matched in a filter. - -### **2.2.3.** Multicast, ARP, NDP, and Special Addressing Modes - -ZeroTier networks support multicast via a simple publish/subscribe system. - -When a node wishes to receive multicasts for a given multicast group, it advertises membership in this group to other members of the network with which it is communicating and to the network controller. When a node wishes to send a multicast it both consults its cache of recent advertisements and periodically solicits additional advertisements. - -Broadcast (Ethernet *ff:ff:ff:ff:ff:ff*) is treated as a multicast group to which all members subscribe. It can be disabled at the network level to reduce traffic if it is not needed. IPv4 ARP receives special handling (see below) and will still work if normal broadcast is disabled. - -Multicasts are propagated using simple sender-side replication. This places the full outbound bandwidth load for multicast on the sender and minimizes multicast latency. Network configurations contain a network-wide **multicast limit** configurable at the network controller. This specifies the maximum number of other nodes to which any node will send a multicast. If the number of known recipients in a given multicast group exceeds the multicast limit, the sender chooses a random subset. - -There is no global limit on multicast recipients, but setting the multicast limit very high on very large networks could result in significant bandwidth overhead. - -#### Special Handling of IPv4 ARP Broadcasts - -IPv4 [ARP](https://en.wikipedia.org/wiki/Address_Resolution_Protocol) is built on simple Ethernet broadcast and scales poorly on large or distributed networks. To improve ARP's scalability ZeroTier generates a unique multicast group for each IPv4 address detected on its system and then transparently intercepts ARP queries and sends them only to the correct group. This converts ARP into effectively a unicast or narrow multicast protocol (like IPv6 NDP) and allows IPv4 ARP to work reliably across wide area networks without excess bandwidth consumption. A similar strategy is implemented under the hood by a number of enterprise switches and WiFi routers designed for deployment on extremely large LANs. This ARP emulation mode is transparent to the OS and application layers, but it does mean that packet sniffers will not see all ARP queries on a virtual network the way they typically can on smaller wired LANs. - -#### Multicast-Free IPv6 Addressing Modes - -IPv6 uses a protocol called [NDP](https://en.wikipedia.org/wiki/Neighbor_Discovery_Protocol) in place of ARP. It is similar in role and design but uses narrow multicast in place of broadcast for superior scalability on large networks. This protocol nevertheless still imposes the latency of an additional multicast lookup whenever a new address is contacted. This can add hundreds of milliseconds over a wide area network, or more if latencies associated with pub/sub recipient lookup are significant. - -IPv6 addresses are large enough to easily encode ZeroTier addresses. For faster operation and better scaling we've implemented several special IPv6 addressing modes that allow the local node to emulate NDP. These are ZeroTier's **rfc4193** and **6plane** IPv6 address assignment schemes. If these addressing schemes are enabled on a network, nodes locally intercept outbound NDP queries for matching addresses and then locally generate spoofed NDP replies. - -Both modes dramatically reduce initial connection latency between network members. **6plane** additionally exploits NDP emulation to transparently assign an entire IPv6 /80 prefix to every node without requiring any node to possess additional routing table entries. This is designed for virtual machine and container hosts that wish to auto-assign IPv6 addresses to guests and is very useful on microservice architecture backplane networks. - -Finally there is a security benefit to NDP emulation. ZeroTier addresses are cryptographically authenticated, and since Ethernet MAC addresses on networks are computed from ZeroTier addresses these are also secure. NDP emulated IPv6 addressing modes are therefore not vulnerable to NDP reply spoofing. - -Normal non-NDP-emulated IPv6 addresses (including link-local addresses) can coexist with NDP-emulated addressing schemes. Any NDP queries that do not match NDP-emulated addresses are sent via normal multicast. - -### **2.2.4.** Ethernet Bridging - -ZeroTier emulates a true Ethernet switch. This includes the ability to L2 bridge other Ethernet networks (wired LAN, WiFi, virtual backplanes, etc.) to virtual networks using conventional Ethernet bridging. - -To act as a bridge a network member must be designated as such by the controller. This is for security reasons as normal network members are not permitted to send traffic from any origin other than their MAC address. Designated bridges also receive special treatment from the multicast algorithm, which more aggressively and directly queries them for group subscriptions and replicates all broadcast traffic and ARP requests to them. As a result bridge nodes experience a slightly higher amount of multicast bandwidth overhead. - -Bridging has been tested extensively on Linux using the Linux kernel native bridge, which cleanly handles network MTU mismatch. There are third party reports of bridging working on other platforms. The details of setting up bridging, including how to selectively block traffic like DHCP that may not be wanted across the bridge, are beyond the scope of this manual. - -### **2.2.5.** Public Networks - -It is possible to disable access control on a ZeroTier network. A public network's members do not check certificates of membership, and new members to a public network are automatically marked as authorized by their host controller. It is not possible to de-authorize a member from a public network. - -Rules on the other hand *are* enforced, so it's possible to implement a special purpose public network that only allows access to a few things or that only allows a restricted subset of traffic. - -Public networks are useful for testing and for peer to peer "party lines" for gaming, chat, and other applications. Participants in public networks are warned to pay special attention to security. If joining a public network be careful not to expose vulnerable services or accidentally share private files via open network shares or HTTP servers. Make sure your operating system, applications, and services are fully up to date. - -ZeroTier, Inc. operates a public network called Earth (no relation to the root server planet definition of the same name) with the network ID `8056c2e21c000001`. Earth issues IPv4 addresses in the unused IPv4 space 28.0.0.0/7 and rfc4193 IPv6 addresses and allows multicast for service discovery. It's essentially a global LAN party. After joining Earth visit `http://earth.zerotier.net/` to get a page showing your Earth virtual IP address and Ethernet MAC address. - -### **2.2.6.** Ad-Hoc Networks - -A special kind of public network called an ad-hoc network may be accessed by joining a network ID with the format: - - ffSSSSEEEE000000 - | | | | - | | | Reserved for future use, must be 0 - | | End of port range (hex) - | Start of port range (hex) - Reserved ZeroTier address prefix indicating a controller-less network - -Ad-hoc networks are public (no access control) networks that have no network controller. Instead their configuration and other credentials are generated locally. Ad-hoc networks permit only IPv6 UDP and TCP unicast traffic (no multicast or broadcast) using 6plane format NDP-emulated IPv6 addresses. In addition an ad-hoc network ID encodes an IP port range. UDP packets and TCP SYN (connection open) packets are only allowed to desintation ports within the encoded range. - -For example `ff00160016000000` is an ad-hoc network allowing only SSH, while `ff0000ffff000000` is an ad-hoc network allowing any UDP or TCP port. - -Keep in mind that these networks are public and anyone in the entire world can join them. Care must be taken to avoid exposing vulnerable services or sharing unwanted files or other resources. - -## **3.** The Network Rules Engine - -Traffic on ZeroTier networks can be observed and controlled with a system of globally applied network rules. These are enforced in a distributed fashion by both the senders and the receivers of packets, meaning that to escape the rules engine a malicious attacker would need to compromise systems on both sides of the conversation. - -The ZeroTier VL2 rules engine differs from most other firewalls and SDN rules engines in several ways. The most immediately relevant of these is that the ZeroTier rules engine is stateless, meaning it lacks connection tracking. This means that bidirectional whitelisting can't be accomplished by simply whitelisting reply packets to established connections. Instead some thought must be put into how to allow both sides of a desired flow. Rule patterns to achieve the most common desired objectives are included in this manual. - -The decision to make our rules engine stateless was a design trade-off driven by several concerns. First we wanted to keep complexity, code footprint, and memory use very low to support small embedded devices. The second and more fundamental reason is that distributed stateful filtering requires distributed state synchronization. This would have added a large volume of additional sync traffic as well as introducing [inescapable](https://en.wikipedia.org/wiki/CAP_theorem) new sources of instablity and failure and a lot of surface area for security vulnerabilities. - -While ZeroTier lacks state tracking, its rules engine includes something not found anywhere else in the enterprise networking space: [capability-based security](https://en.wikipedia.org/wiki/Capability-based_security) and device tagging. Capabilities and tags allow extremely complex micro-segmented network rule schemes to be implemented in a sane, conceptual way that is both easier for human beings to understand and more efficient for machines to handle. - -This section assumes some level of familiarity with network rules as they're commonly used on firewalls and routers, etc. While the rules engine is part of VL2, it's been given its own section in this manual due to the depth and cross-cutting nature of the topic. - -### **3.1.** Rule Sets and Rule Evaluation - -Rule sets are ordered lists of one or more rules, with each rule consisting of one or more **match** conditions followed by one **action**. As a rule set is evaluated, each match is tested in order and is then ANDed or ORed with the previous match result state. When an action is encountered it is taken if the result of the preceding matches is *true*. An action with no preceding matches is always taken. If no permissive actions are taken by any rule set the packet is discarded. - -Here is a simple rule set that constrains Ethernet traffic on a network to only IPv4, ARP, or IPv6 as it would appear in the raw JSON format used by ZeroTier One's built-in network controller implementation. Don't worry if this seems verbose and difficult. We have a more human-friendly way of writing rule sets, but before we introduce it it's important to understand what is really happening. - - [ - { - "etherType": 2048, - "not": true, - "or": false, - "type": "MATCH_ETHERTYPE" - }, - { - "etherType": 2054, - "not": true, - "or": false, - "type": "MATCH_ETHERTYPE" - }, - { - "etherType": 34525, - "not": true, - "or": false, - "type": "MATCH_ETHERTYPE" - }, - { - "type": "ACTION_DROP" - }, - { - "type": "ACTION_ACCEPT" - } - ] - -This checks whether an Ethernet level packet is _not_ IPv4 (ethertype 2048) _and not_ IPv4 ARP (ethertype 2054) _and not_ IPv6 (ethertype 34525). If all three matches evaluate to true (meaning the ethertype is none of these) then the **drop** action is taken. Otherwise the **accept** action is taken. - -Networks have one base rule set that is applied to all traffic. Its size is constrained to 1024 entries (each match or action is an entry). It should be used to set the overall policies for all members of the network, and for most common use cases it's all you'll need. For more complex scenarios, both capabilities and tags provide methods of both managing complexity and scaling the overall size of a network's rule system. - -### **3.2.** Capabilities - -A capability is a small rule set that is bundled into a credential object, signed by the network controller, and issued to only those member(s) permitted to exercise it. When a member detects that outgoing traffic does not match the base rule set but is allowed by one of its capabilities, it periodically pushes the matching capability credential to the recipient ahead of the packet(s) in question. Peer to peer capability distribution is automatic and is triggered by capability match. - -When the recipient receives the capability it authenticates it by checking its signature and timestamp and, provided the capability is valid, adds it to the set of capabilities to apply to incoming traffic from the capability's owner. The sender has effectively told the recipient "I can too send this packet! Teacher says so!" - -Capabilities allow large systems of rules to be broken down into functional aspects and then distributed intelligently only to those members with a need to know. This avoids the bandwidth and storage overhead of distributing huge monolithic rule sets and organizes rules conceptually to make them easier for administrators to understand. - -There are three terminating actions that can be taken in a rule set: **accept**, **break**, and **drop**. The accept action terminates rule evaluation and accepts the packet. The break action terminates the evaluation of the current rule set but permits the further evaluation of capabilities. The drop action terminates rule evaluation and drops the packet without checking capabilities in the base rule set, but is equivalent to break in capability rule sets. In most cases break should be used unless certain traffic must be absolutely prohibited under any circumstance. - -In the simple base rule set example in section 3.1 the drop action is taken in the unapproved case. This means that ethernet whitelisting cannot be overridden by a capability. If we change `ACTION_DROP` in our example to `ACTION_BREAK`, then it becomes possible to issue the following capability: - - [ - { - "etherType": 2114, - "not": false, - "or": false, - "type": "MATCH_ETHERTYPE" - }, - { - "type": "ACTION_ACCEPT" - } - ] - -Ethertype 2114 is [wake-on-LAN](https://wiki.wireshark.org/WakeOnLAN), a special packet that can cause some systems to wake from sleep mode. If we place the above tiny rule set into a capability and issue it to a device, this device *but no others* will now be permitted to send wake-on-LAN magic packets. (Wake-on-LAN requires hardware support so it would only work to target devices plugged into a physical network bridged to a ZeroTier network, but don't worry about that here. It's just an example of special traffic.) - -Capability rule sets are limited to only 64 entries. The idea is to keep them small and simple. A capability should grant one thing or one small set of conceptually related things. - -### **3.3.** Tags - -ZeroTier provides a second mechanism to control rule set complexity. Tags are 32-bit numeric key-value pair credentials that are issued to network members and signed by the controller. They are then distributed peer to peer on a need to know basis in a similar manner to capabilities. - -Tags provide a way to conditionally drop or allow traffic between members by member classification. They allow very detailed network micro-segmentation by member role, permission, function, etc. without resulting in a combinatorial explosion in rules table size. - -Let's say we want to permit traffic on TCP ports 139 and 445 (netbios/CIFS file sharing) only between systems that belong to the same department. Our company has 12,000 devices and 10 departments. Without tags this would require 144,000,000 rules, but with tags it can be accomplished by only a few. - -First a tag is created to represent the department. Let's give it tag ID 100. Each member system receives the tag with a value from 1 to 10 indicating which department it belongs to. We can then add the following rules to our network's base rule set (or to a capability if so desired): - - [ - { - "type": "MATCH_IP_DEST_PORT_RANGE", - "not": false, - "or": false, - "start": 139, - "end": 139 - }, - { - "type": "MATCH_IP_DEST_PORT_RANGE", - "not": false, - "or": true, - "start": 445, - "end": 445 - }, - { - "type": "MATCH_IP_PROTOCOL", - "not": false, - "or": false, - "ipProtocol": 6 - }, - { - "type": "MATCH_TAGS_DIFFERENCE", - "not": false, - "or": false, - "id": 10, - "value": 0 - }, - { - "type": "ACTION_ACCEPT" - } - ] - -This tells members in our network to accept TCP packets on ports 139 or 445 if the difference between tags with tag ID 10 is zero, meaning they match. (If a member does not have a value for this tag, it does not match.) Now all members of the same department can access CIFS file shares, but CIFS sharing between departments could still be prohibited. (TCP whitelisting requires some additional rules due to the stateless nature of our rules engine. See the section below on rule design patterns.) - -Tags can be compared on numeric value or as bit fields via several different bit mask operations allowing many different systems of member classification to be implemented. - -### **3.4.** Rule Description Language - -- cgit v1.2.3 From d7b4f24a7a8193992c39140fdc6c40a58f3fb2b3 Mon Sep 17 00:00:00 2001 From: Adam Ierymenko Date: Thu, 1 Jun 2017 17:21:57 -0700 Subject: . --- attic/kubernetes_docs/.zerotierCliSettings | 18 ++++ attic/kubernetes_docs/Dockerfile | 19 ++++ attic/kubernetes_docs/README.md | 150 +++++++++++++++++++++++++++++ attic/kubernetes_docs/entrypoint.sh | 23 +++++ attic/kubernetes_docs/server.js | 8 ++ doc/ext/kubernetes/.zerotierCliSettings | 18 ---- doc/ext/kubernetes/Dockerfile | 19 ---- doc/ext/kubernetes/README.md | 150 ----------------------------- doc/ext/kubernetes/entrypoint.sh | 23 ----- doc/ext/kubernetes/server.js | 8 -- 10 files changed, 218 insertions(+), 218 deletions(-) create mode 100644 attic/kubernetes_docs/.zerotierCliSettings create mode 100644 attic/kubernetes_docs/Dockerfile create mode 100644 attic/kubernetes_docs/README.md create mode 100644 attic/kubernetes_docs/entrypoint.sh create mode 100644 attic/kubernetes_docs/server.js delete mode 100644 doc/ext/kubernetes/.zerotierCliSettings delete mode 100644 doc/ext/kubernetes/Dockerfile delete mode 100644 doc/ext/kubernetes/README.md delete mode 100644 doc/ext/kubernetes/entrypoint.sh delete mode 100644 doc/ext/kubernetes/server.js (limited to 'doc') diff --git a/attic/kubernetes_docs/.zerotierCliSettings b/attic/kubernetes_docs/.zerotierCliSettings new file mode 100644 index 00000000..0e7df9b6 --- /dev/null +++ b/attic/kubernetes_docs/.zerotierCliSettings @@ -0,0 +1,18 @@ +{ + "configVersion": 1, + "defaultCentral": "@my.zerotier.com", + "defaultController": "@my.zerotier.com", + "defaultOne": "@local", + "things": { + "local": { + "auth": "local_service_auth_token_replaced_automatically", + "type": "one", + "url": "http://127.0.0.1:9993/" + }, + "my.zerotier.com": { + "auth": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", + "type": "central", + "url": "https://my.zerotier.com/" + } + } +} diff --git a/attic/kubernetes_docs/Dockerfile b/attic/kubernetes_docs/Dockerfile new file mode 100644 index 00000000..6437a2bb --- /dev/null +++ b/attic/kubernetes_docs/Dockerfile @@ -0,0 +1,19 @@ +FROM node:4.4 +EXPOSE 8080/tcp 9993/udp + +# Install ZT network conf files +RUN mkdir -p /var/lib/zerotier-one/networks.d +ADD *.conf /var/lib/zerotier-one/networks.d/ +ADD *.conf / +ADD zerotier-one / +ADD zerotier-cli / +ADD .zerotierCliSettings / + +# Install App +ADD server.js / + +# script which will start/auth VM on ZT network +ADD entrypoint.sh / +RUN chmod -v +x /entrypoint.sh + +CMD ["./entrypoint.sh"] \ No newline at end of file diff --git a/attic/kubernetes_docs/README.md b/attic/kubernetes_docs/README.md new file mode 100644 index 00000000..482e77e5 --- /dev/null +++ b/attic/kubernetes_docs/README.md @@ -0,0 +1,150 @@ +Kubernetes + ZeroTier +==== + +A self-authorizing Kubernetes cluster deployment over a private ZeroTier network. + +This is a quick tutorial for setting up a Kubernetes deployment which can self-authorize each new replica onto your private ZeroTier network with no additional configuration needed when you scale. The Kubernetes-specific instructions and content is based on the [hellonode](http://kubernetes.io/docs/hellonode/) tutorial. All of the files discussed below can be found [here](); + + + +## Preliminary tasks + +**Step 1: Go to [my.zerotier.com](https://my.zerotier.com) and generate a network controller API key. This key will be used by ZeroTier to automatically authorize new instances of your VMs to join your secure deployment network during replication.** + +**Step 2: Create a new `private` network. Take note of the network ID, henceforth: `nwid`** + +**Step 3: Follow the instructions from the [hellonode](ttp://kubernetes.io/docs/hellonode/) tutorial to set up your development system.** + +*** +## Construct docker image + +**Step 4: Create necessary files for inclusion into image, your resultant directory should contain:** + + - `ztkube/.conf` + - `ztkube/Dockerfile` + - `ztkube/entrypoint.sh` + - `ztkube/server.js` + - `ztkube/zerotier-cli` + - `ztkube/zerotier-one` + +Start by creating a build directory to copy all required files into `mkdir ztkube`. Then build the following: + - `make one` + - `make cli` + +Add the following files to the `ztkube` directory. These files will be compiled into the Docker image. + + - Create an empty `.conf` file to specify the private deployment network you created in *Step 2*: + + - Create a CLI tool config file `.zerotierCliSettings` which should only contain your network controller API key to authorize new devices on your network (the local service API key will be filled in automatically). In this example the default controller is hosted by us at [my.zerotier.com](https://my.zerotier.com). Alternatively, you can host your own network controller but you'll need to modify the CLI config file accordingly. + +``` +{ + "configVersion": 1, + "defaultCentral": "@my.zerotier.com", + "defaultController": "@my.zerotier.com", + "defaultOne": "@local", + "things": { + "local": { + "auth": "local_service_auth_token_replaced_automatically", + "type": "one", + "url": "http://127.0.0.1:9993/" + }, + "my.zerotier.com": { + "auth": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", + "type": "central", + "url": "https://my.zerotier.com/" + } + } +} +``` + + + - Create a `Dockerfile` which will copy the ZeroTier service as well as the ZeroTier CLI to the image: + +``` +FROM node:4.4 +EXPOSE 8080/tcp 9993/udp + +# Install ZT network conf files +RUN mkdir -p /var/lib/zerotier-one/networks.d +ADD *.conf /var/lib/zerotier-one/networks.d/ +ADD *.conf / +ADD zerotier-one / +ADD zerotier-cli / +ADD .zerotierCliSettings / + +# Install App +ADD server.js / + +# script which will start/auth VM on ZT network +ADD entrypoint.sh / +RUN chmod -v +x /entrypoint.sh + +CMD ["./entrypoint.sh"] +``` + + - Create the `entrypoint.sh` script which will start the ZeroTier service in the VM, attempt to join your deployment network and automatically authorize the new VM if your network is set to private: + +``` +#!/bin/bash + +echo '*** ZeroTier-Kubernetes self-auth test script' +chown -R daemon /var/lib/zerotier-one +chgrp -R daemon /var/lib/zerotier-one +su daemon -s /bin/bash -c '/zerotier-one -d -U -p9993 >>/tmp/zerotier-one.out 2>&1' +dev="" +nwconf=$(ls *.conf) +nwid="${nwconf%.*}" + +sleep 10 +dev=$(cat /var/lib/zerotier-one/identity.public| cut -d ':' -f 1) + +echo '*** Joining' +./zerotier-cli join "$nwid".conf +# Fill out local service auth token +AUTHTOKEN=$(cat /var/lib/zerotier-one/authtoken.secret) +sed "s|\local_service_auth_token_replaced_automatically|${AUTHTOKEN}|" .zerotierCliSettings > /root/.zerotierCliSettings +echo '*** Authorizing' +./zerotier-cli net-auth @my.zerotier.com "$nwid" "$dev" +echo '*** Cleaning up' # Remove controller auth token +rm -rf .zerotierCliSettings /root/.zerotierCliSettings +node server.js +``` + +**Step 5: Build the image:** + + - `docker build -t gcr.io/$PROJECT_ID/hello-node .` + + + +**Step 6: Push the docker image to your *Container Registry*** + + - `gcloud docker push gcr.io/$PROJECT_ID/hello-node:v1` + +*** +## Deploy! + +**Step 7: Create Kubernetes Cluster** + + - `gcloud config set compute/zone us-central1-a` + + - `gcloud container clusters create hello-world` + + - `gcloud container clusters get-credentials hello-world` + + + +**Step 8: Create your pod** + + - `kubectl run hello-node --image=gcr.io/$PROJECT_ID/hello-node:v1 --port=8080` + + + +**Step 9: Scale** + + - `kubectl scale deployment hello-node --replicas=4` + +*** +## Verify + +Now, after a minute or so you can use `zerotier-cli net-members ` to show all of your VM instances on your ZeroTier deployment network. If you haven't [configured your local CLI](https://github.com/zerotier/ZeroTierOne/tree/dev/cli), you can simply log into [my.zerotier.com](https://my.zerotier.com), go to *Networks -> nwid* to check that your VMs are indeed members of your private network. You should also note that the `entrypoint.sh` script will automatically delete your network controller API key once it has authorized your VM. This is merely a security measure and can be removed if needed. diff --git a/attic/kubernetes_docs/entrypoint.sh b/attic/kubernetes_docs/entrypoint.sh new file mode 100644 index 00000000..80cd278e --- /dev/null +++ b/attic/kubernetes_docs/entrypoint.sh @@ -0,0 +1,23 @@ +#!/bin/bash + +echo '*** ZeroTier-Kubernetes self-auth test script' +chown -R daemon /var/lib/zerotier-one +chgrp -R daemon /var/lib/zerotier-one +su daemon -s /bin/bash -c '/zerotier-one -d -U -p9993 >>/tmp/zerotier-one.out 2>&1' +dev="" +nwconf=$(ls *.conf) +nwid="${nwconf%.*}" + +sleep 10 +dev=$(cat /var/lib/zerotier-one/identity.public| cut -d ':' -f 1) + +echo '*** Joining' +./zerotier-cli join "$nwid".conf +# Fill out local service auth token +AUTHTOKEN=$(cat /var/lib/zerotier-one/authtoken.secret) +sed "s|\local_service_auth_token_replaced_automatically|${AUTHTOKEN}|" .zerotierCliSettings > /root/.zerotierCliSettings +echo '*** Authorizing' +./zerotier-cli net-auth @my.zerotier.com "$nwid" "$dev" +echo '*** Cleaning up' # Remove controller auth token +rm -rf .zerotierCliSettings /root/.zerotierCliSettings +node server.js \ No newline at end of file diff --git a/attic/kubernetes_docs/server.js b/attic/kubernetes_docs/server.js new file mode 100644 index 00000000..a4b08bb8 --- /dev/null +++ b/attic/kubernetes_docs/server.js @@ -0,0 +1,8 @@ +var http = require('http'); +var handleRequest = function(request, response) { + console.log('Received request for URL: ' + request.url); + response.writeHead(200); + response.end('Hello World!'); +}; +var www = http.createServer(handleRequest); +www.listen(8080); diff --git a/doc/ext/kubernetes/.zerotierCliSettings b/doc/ext/kubernetes/.zerotierCliSettings deleted file mode 100644 index 0e7df9b6..00000000 --- a/doc/ext/kubernetes/.zerotierCliSettings +++ /dev/null @@ -1,18 +0,0 @@ -{ - "configVersion": 1, - "defaultCentral": "@my.zerotier.com", - "defaultController": "@my.zerotier.com", - "defaultOne": "@local", - "things": { - "local": { - "auth": "local_service_auth_token_replaced_automatically", - "type": "one", - "url": "http://127.0.0.1:9993/" - }, - "my.zerotier.com": { - "auth": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", - "type": "central", - "url": "https://my.zerotier.com/" - } - } -} diff --git a/doc/ext/kubernetes/Dockerfile b/doc/ext/kubernetes/Dockerfile deleted file mode 100644 index 6437a2bb..00000000 --- a/doc/ext/kubernetes/Dockerfile +++ /dev/null @@ -1,19 +0,0 @@ -FROM node:4.4 -EXPOSE 8080/tcp 9993/udp - -# Install ZT network conf files -RUN mkdir -p /var/lib/zerotier-one/networks.d -ADD *.conf /var/lib/zerotier-one/networks.d/ -ADD *.conf / -ADD zerotier-one / -ADD zerotier-cli / -ADD .zerotierCliSettings / - -# Install App -ADD server.js / - -# script which will start/auth VM on ZT network -ADD entrypoint.sh / -RUN chmod -v +x /entrypoint.sh - -CMD ["./entrypoint.sh"] \ No newline at end of file diff --git a/doc/ext/kubernetes/README.md b/doc/ext/kubernetes/README.md deleted file mode 100644 index 482e77e5..00000000 --- a/doc/ext/kubernetes/README.md +++ /dev/null @@ -1,150 +0,0 @@ -Kubernetes + ZeroTier -==== - -A self-authorizing Kubernetes cluster deployment over a private ZeroTier network. - -This is a quick tutorial for setting up a Kubernetes deployment which can self-authorize each new replica onto your private ZeroTier network with no additional configuration needed when you scale. The Kubernetes-specific instructions and content is based on the [hellonode](http://kubernetes.io/docs/hellonode/) tutorial. All of the files discussed below can be found [here](); - - - -## Preliminary tasks - -**Step 1: Go to [my.zerotier.com](https://my.zerotier.com) and generate a network controller API key. This key will be used by ZeroTier to automatically authorize new instances of your VMs to join your secure deployment network during replication.** - -**Step 2: Create a new `private` network. Take note of the network ID, henceforth: `nwid`** - -**Step 3: Follow the instructions from the [hellonode](ttp://kubernetes.io/docs/hellonode/) tutorial to set up your development system.** - -*** -## Construct docker image - -**Step 4: Create necessary files for inclusion into image, your resultant directory should contain:** - - - `ztkube/.conf` - - `ztkube/Dockerfile` - - `ztkube/entrypoint.sh` - - `ztkube/server.js` - - `ztkube/zerotier-cli` - - `ztkube/zerotier-one` - -Start by creating a build directory to copy all required files into `mkdir ztkube`. Then build the following: - - `make one` - - `make cli` - -Add the following files to the `ztkube` directory. These files will be compiled into the Docker image. - - - Create an empty `.conf` file to specify the private deployment network you created in *Step 2*: - - - Create a CLI tool config file `.zerotierCliSettings` which should only contain your network controller API key to authorize new devices on your network (the local service API key will be filled in automatically). In this example the default controller is hosted by us at [my.zerotier.com](https://my.zerotier.com). Alternatively, you can host your own network controller but you'll need to modify the CLI config file accordingly. - -``` -{ - "configVersion": 1, - "defaultCentral": "@my.zerotier.com", - "defaultController": "@my.zerotier.com", - "defaultOne": "@local", - "things": { - "local": { - "auth": "local_service_auth_token_replaced_automatically", - "type": "one", - "url": "http://127.0.0.1:9993/" - }, - "my.zerotier.com": { - "auth": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", - "type": "central", - "url": "https://my.zerotier.com/" - } - } -} -``` - - - - Create a `Dockerfile` which will copy the ZeroTier service as well as the ZeroTier CLI to the image: - -``` -FROM node:4.4 -EXPOSE 8080/tcp 9993/udp - -# Install ZT network conf files -RUN mkdir -p /var/lib/zerotier-one/networks.d -ADD *.conf /var/lib/zerotier-one/networks.d/ -ADD *.conf / -ADD zerotier-one / -ADD zerotier-cli / -ADD .zerotierCliSettings / - -# Install App -ADD server.js / - -# script which will start/auth VM on ZT network -ADD entrypoint.sh / -RUN chmod -v +x /entrypoint.sh - -CMD ["./entrypoint.sh"] -``` - - - Create the `entrypoint.sh` script which will start the ZeroTier service in the VM, attempt to join your deployment network and automatically authorize the new VM if your network is set to private: - -``` -#!/bin/bash - -echo '*** ZeroTier-Kubernetes self-auth test script' -chown -R daemon /var/lib/zerotier-one -chgrp -R daemon /var/lib/zerotier-one -su daemon -s /bin/bash -c '/zerotier-one -d -U -p9993 >>/tmp/zerotier-one.out 2>&1' -dev="" -nwconf=$(ls *.conf) -nwid="${nwconf%.*}" - -sleep 10 -dev=$(cat /var/lib/zerotier-one/identity.public| cut -d ':' -f 1) - -echo '*** Joining' -./zerotier-cli join "$nwid".conf -# Fill out local service auth token -AUTHTOKEN=$(cat /var/lib/zerotier-one/authtoken.secret) -sed "s|\local_service_auth_token_replaced_automatically|${AUTHTOKEN}|" .zerotierCliSettings > /root/.zerotierCliSettings -echo '*** Authorizing' -./zerotier-cli net-auth @my.zerotier.com "$nwid" "$dev" -echo '*** Cleaning up' # Remove controller auth token -rm -rf .zerotierCliSettings /root/.zerotierCliSettings -node server.js -``` - -**Step 5: Build the image:** - - - `docker build -t gcr.io/$PROJECT_ID/hello-node .` - - - -**Step 6: Push the docker image to your *Container Registry*** - - - `gcloud docker push gcr.io/$PROJECT_ID/hello-node:v1` - -*** -## Deploy! - -**Step 7: Create Kubernetes Cluster** - - - `gcloud config set compute/zone us-central1-a` - - - `gcloud container clusters create hello-world` - - - `gcloud container clusters get-credentials hello-world` - - - -**Step 8: Create your pod** - - - `kubectl run hello-node --image=gcr.io/$PROJECT_ID/hello-node:v1 --port=8080` - - - -**Step 9: Scale** - - - `kubectl scale deployment hello-node --replicas=4` - -*** -## Verify - -Now, after a minute or so you can use `zerotier-cli net-members ` to show all of your VM instances on your ZeroTier deployment network. If you haven't [configured your local CLI](https://github.com/zerotier/ZeroTierOne/tree/dev/cli), you can simply log into [my.zerotier.com](https://my.zerotier.com), go to *Networks -> nwid* to check that your VMs are indeed members of your private network. You should also note that the `entrypoint.sh` script will automatically delete your network controller API key once it has authorized your VM. This is merely a security measure and can be removed if needed. diff --git a/doc/ext/kubernetes/entrypoint.sh b/doc/ext/kubernetes/entrypoint.sh deleted file mode 100644 index 80cd278e..00000000 --- a/doc/ext/kubernetes/entrypoint.sh +++ /dev/null @@ -1,23 +0,0 @@ -#!/bin/bash - -echo '*** ZeroTier-Kubernetes self-auth test script' -chown -R daemon /var/lib/zerotier-one -chgrp -R daemon /var/lib/zerotier-one -su daemon -s /bin/bash -c '/zerotier-one -d -U -p9993 >>/tmp/zerotier-one.out 2>&1' -dev="" -nwconf=$(ls *.conf) -nwid="${nwconf%.*}" - -sleep 10 -dev=$(cat /var/lib/zerotier-one/identity.public| cut -d ':' -f 1) - -echo '*** Joining' -./zerotier-cli join "$nwid".conf -# Fill out local service auth token -AUTHTOKEN=$(cat /var/lib/zerotier-one/authtoken.secret) -sed "s|\local_service_auth_token_replaced_automatically|${AUTHTOKEN}|" .zerotierCliSettings > /root/.zerotierCliSettings -echo '*** Authorizing' -./zerotier-cli net-auth @my.zerotier.com "$nwid" "$dev" -echo '*** Cleaning up' # Remove controller auth token -rm -rf .zerotierCliSettings /root/.zerotierCliSettings -node server.js \ No newline at end of file diff --git a/doc/ext/kubernetes/server.js b/doc/ext/kubernetes/server.js deleted file mode 100644 index a4b08bb8..00000000 --- a/doc/ext/kubernetes/server.js +++ /dev/null @@ -1,8 +0,0 @@ -var http = require('http'); -var handleRequest = function(request, response) { - console.log('Received request for URL: ' + request.url); - response.writeHead(200); - response.end('Hello World!'); -}; -var www = http.createServer(handleRequest); -www.listen(8080); -- cgit v1.2.3