webproxy: update docs

1 files changed, 368 insertions, 91 deletions

diff --git a/docs/configuration/service/webproxy.rst b/docs/configuration/service/webproxy.rst
index e65c672c..e8f6423e 100644
--- a/docs/configuration/service/webproxy.rst
+++ b/docs/configuration/service/webproxy.rst
@@ -1,137 +1,387 @@
-Webproxy
---------
+.. _webproxy:
 
-The proxy service in VyOS is based on Squid3 and some related modules.
+########
+Webproxy
+########
 
-Squid3_ is a caching and forwarding HTTP web proxy. It has a wide variety of
-uses, including speeding up a web server by caching repeated requests,
-caching web, DNS and other computer network lookups for a group of people
-sharing network resources, and aiding security by filtering traffic. Although
-primarily used for HTTP and FTP, Squid includes limited support for several
-other protocols including Internet Gopher, SSL,[6] TLS and HTTPS. Squid does
-not support the SOCKS protocol.
+The proxy service in VyOS is based on Squid_ and some related modules.
 
-All examples here assumes that your inside ip address is ``192.168.0.1``.
-Replace with your own where applicable.
+Squid_ is a caching and forwarding HTTP web proxy. It has a wide variety of
+uses, including speeding up a web server by caching repeated requests, caching
+web, DNS and other computer network lookups for a group of people sharing
+network resources, and aiding security by filtering traffic. Although primarily
+used for HTTP and FTP, Squid includes limited support for several other
+protocols including Internet Gopher, SSL,[6] TLS and HTTPS. Squid does not
+support the SOCKS protocol.
 
-URL Filtering is provided by Squidguard_.
+URL Filtering is provided by SquidGuard_.
 
+*************
 Configuration
-^^^^^^^^^^^^^^
+*************
 
-.. code-block:: none
+.. cfgcmd:: set service webproxy append-domain <domain>
 
-  # Enable proxy service
-  set service webproxy listen-address 192.168.0.1
+  Use this command to specify a domain name to be appended to domain-names
+  within URLs that do not include a dot ``.`` the domain is appended.
 
-  # By default it will listen to port 3128. If you want something else you have to define that.
-  set service webproxy listen-address 192.168.0.1 port 2050
+  Example: to be appended is set to ``vyos.net`` and the URL received is
+  ``www/foo.html``, the system will use the generated, final URL of
+  ``www.vyos.net/foo.html``.
 
-  # By default the transparent proxy on that interface is enabled. To disable that you simply
-  set service webproxy listen-address 192.168.0.1 disable-transparent
+  .. code-block:: none
 
-  # Block specific urls
-  set service webproxy url-filtering squidguard local-block myspace.com
+    set service webproxy append-domain vyos.net
 
-  # If you want to you can log these blocks
-  set service webproxy url-filtering squidguard log local-block
+.. cfgcmd:: set service webproxy cache-size <size>
 
+  The size of the on-disk Proxy cache is user configurable. The Proxies default
+  cache-size is configured to 100 MB.
 
-Options
-*******
+  Unit of this command is MB.
 
-Filtering by category
-^^^^^^^^^^^^^^^^^^^^^
+  .. code-block:: none
 
-If you want to use existing blacklists you have to create/download a database
-first. Otherwise you will not be able to commit the config changes.
+    set service webproxy cache-size 1024
 
-.. code-block:: none
+.. cfgcmd:: set service webproxy default-port <port>
 
-  vyos@vyos# commit
-  [ service webproxy ]
-  Warning: no blacklists installed
-  Unknown block-category [ads] for policy [default]
+  Specify the port used on which the proxy service is listening for requests.
+  This port is the default port used for the specified listen-address.
 
-  [[service webproxy]] failed
-  Commit failed
+  Default port is 3128.
 
-* Download/Update complete blacklist
+  .. code-block:: none
 
-  :code:`update webproxy blacklists`
+    set service webproxy default-port 8080
 
-* Download/Update partial blacklist
+.. cfgcmd:: set service webproxy domain-block <domain>
 
-  :code:`update webproxy blacklists category ads`
+  Used to block specific domains by the Proxy. Specifying "vyos.net" will block
+  all access to vyos.net, and specifying ".xxx" will block all access to URLs
+  having an URL ending on .xxx.
 
-  Use tab completion to get a list of categories.
+  .. code-block:: none
 
-* To auto update the blacklist files
+    set service webproxy domain-block vyos.net
 
-  :code:`set service webproxy url-filtering squidguard auto-update
-  update-hour 23`
+.. cfgcmd:: set service webproxy domain-noncache <domain>
 
-* To configure blocking add the following to the configuration
+  Allow access to sites in a domain without retrieving them from the Proxy
+  cache. Specifying "vyos.net" will allow access to vyos.net but the pages
+  accessed will not be cached. It useful for working around problems with
+  "If-Modified-Since" checking at certain sites.
 
-  :code:`set service webproxy url-filtering squidguard block-category ads`
+  .. code-block:: none
 
-  :code:`set service webproxy url-filtering squidguard block-category malware`
+    set service webproxy domain-noncache vyos.net
+
+.. cfgcmd:: set service webproxy listen-address <address>
+
+  Specifies proxy service listening address. The listen address is the IP
+  address on which the web proxy service listens for client requests.
+
+  For security, the listen address should only be used on internal/trusted
+  networks!
+
+  .. code-block:: none
+
+    set service webproxy listen-address 192.0.2.1
+
+.. cfgcmd:: set service webproxy listen-address <address> disable-transparent
+
+  Disables web proxy transparent mode at a listening address.
+
+  In transparent proxy mode, all traffic arriving on port 80 and destined for
+  the Internet is automatically forwarded through the proxy. This allows
+  immediate proxy forwarding without configuring client browsers.
+
+  Non-transparent proxying requires that the client browsers be configured with
+  the proxy settings before requests are redirected. The advantage of this is
+  that the client web browser can detect that a proxy is in use and can behave
+  accordingly. In addition, web-transmitted malware can sometimes be blocked by
+  a non-transparent web proxy, since they are not aware of the proxy settings.
+
+  .. code-block:: none
+
+    set service webproxy listen-address 192.0.2.1 disable-transparent
+
+.. cfgcmd:: set service webproxy listen-address <address> port <port>
+
+  Sets the listening port for a listening address. This overrides the default
+  port of 3128 on the specific listen address.
+
+  .. code-block:: none
+
+    set service webproxy listen-address 192.0.2.1 port 8080
+
+
+.. cfgcmd:: set service webproxy reply-block-mime <mime>
+
+  Used to block a specific mime-type.
+
+  .. code-block:: none
+
+    # block all PDFs
+    set service webproxy reply-block-mime application/pdf
+
+
+.. cfgcmd:: set service webproxy reply-body-max-size <size>
+
+  Specifies the maximum size of a reply body in KB, used to limit the reply
+  size.
+
+  All reply sizes are accepted by default.
+
+  .. code-block:: none
+
+    set service webproxy reply-body-max-size 2048
 
 Authentication
-^^^^^^^^^^^^^^
+==============
 
 The embedded Squid proxy can use LDAP to authenticate users against a company
 wide directory. The following configuration is an example of how to use Active
 Directory as authentication backend. Queries are done via LDAP.
 
-.. code-block:: none
+.. cfgcmd:: set service webproxy authentication children <number>
 
-  vyos@vyos# show service webproxy
-   authentication {
-       children 5
-       credentials-ttl 60
-       ldap {
-           base-dn DC=example,DC=local
-           bind-dn CN=proxyuser,CN=Users,DC=example,DC=local
-           filter-expression (cn=%s)
-           password Qwert1234
-           server ldap.example.local
-           username-attribute cn
-       }
-       method ldap
-       realm "VyOS Webproxy"
-   }
-   cache-size 100
-   default-port 3128
-   listen-address 192.168.188.103 {
-       disable-transparent
-   }
+  Maximum number of authenticator processes to spawn. If you start too few
+  Squid will have to wait for them to process a backlog of credential
+  verifications, slowing it down. When password verifications are done via a
+  (slow) network you are likely to need lots of authenticator processes.
 
-* ``base-dn`` set the base directory for the search
-* ``bind-dn`` and ``password``: set the user, which is used for the ldap search
-* ``filter-expression``: set the exact filter which a authorized user match in
-  a ldap-search. In this example every User is able to authorized.
+  This defaults to 5.
 
-You can find more about the ldap authentication
-`here 
-<http://www.squid-cache.org/Versions/v3/3.2/manuals/basic_ldap_auth.html>`_
+  .. code-block:: none
 
-Adjusting cache size
-^^^^^^^^^^^^^^^^^^^^
+    set service webproxy authentication children 10
 
-The size of the proxy cache can be adjusted by the user.
+.. cfgcmd:: set service webproxy authentication credentials-ttl <time>
 
-.. code-block:: none
+  Specifies how long squid assumes an externally validated username:password
+  pair is valid for - in other words how often the helper program is called for
+  that user. Set this low to force revalidation with short lived passwords.
+
+  Time is in minutes and defaults to 60.
+
+  .. code-block:: none
+
+    set service webproxy authentication credentials-ttl 120
+
+
+.. cfgcmd:: set service webproxy authentication method <ldap>
+
+  Proxy authentication method, currently only LDAP is supported.
+
+  .. code-block:: none
+
+    set service webproxy authentication method ldap
+
+.. cfgcmd:: set service webproxy authentication realm
+
+  Specifies the protection scope (aka realm name) which is to be reported to
+  the client for the authentication scheme. It is commonly part of the text
+  the user will see when prompted for their username and password.
+
+  .. code-block:: none
+
+    set service webproxy authentication realm "VyOS proxy auth"
+
+LDAP
+----
+
+.. cfgcmd:: set service webproxy authentication ldap base-dn <base-dn>
+
+  Specifies the base DN under which the users are located.
+
+  .. code-block:: none
+
+    set service webproxy authentication ldap base-dn DC=vyos,DC=net
+
+
+.. cfgcmd:: set service webproxy authentication ldap bind-dn <bind-dn>
+
+  The DN and password to bind as while performing searches.
+
+  .. code-block:: none
+
+    set service webproxy authentication ldap bind-dn CN=proxyuser,CN=Users,DC=vyos,DC=net
+
+.. cfgcmd:: set service webproxy authentication ldap filter-expression <expr>
+
+  LDAP search filter to locate the user DN. Required if the users are in a
+  hierarchy below the base DN, or if the login name is not what builds the user
+  specific part of the users DN.
+
+  The search filter can contain up to 15 occurrences of %s which will be
+  replaced by the username, as in "uid=%s" for :rfc:`2037` directories. For a
+  detailed description of LDAP search filter syntax see :rfc:`2254`.
+
+  .. code-block:: none
+
+    set service webproxy authentication ldap filter-expression (cn=%s)
+
+.. cfgcmd:: set service webproxy authentication ldap password <password>
+
+  The DN and password to bind as while performing searches. As the password
+  needs to be printed in plain text in your Squid configuration it is strongly
+  recommended to use a account with minimal associated privileges. This to limit
+  the damage in case someone could get hold of a copy of your Squid
+  configuration file.
+
+  .. code-block:: none
 
-  set service webproxy cache-size
-   Possible completions:
-     <0-4294967295>
-                  Disk cache size in MB (default 100)
-     0            Disable disk caching
-     100
+    set service webproxy authentication ldap password vyos
+
+.. cfgcmd:: set service webproxy authentication ldap persistent-connection
+
+  Use a persistent LDAP connection. Normally the LDAP connection is only open
+  while validating a username to preserve resources at the LDAP server. This
+  option causes the LDAP connection to be kept open, allowing it to be reused
+  for further user validations.
+
+  Recommended for larger installations.
+
+  .. code-block:: none
+
+    set service webproxy authentication ldap persistent-connection
+
+.. cfgcmd:: set service webproxy authentication ldap port <port>
+
+  Specify an alternate TCP port where the ldap server is listening if other than
+  the default LDAP port 389.
+
+  .. code-block:: none
+
+    set service webproxy authentication ldap port 389
+
+.. cfgcmd:: set service webproxy authentication ldap server <server>
+
+  Specify the LDAP server to connect to.
+
+  .. code-block:: none
+
+    set service webproxy authentication ldap server ldap.vyos.net
+
+
+.. cfgcmd:: set service webproxy authentication ldap use-ssl
+
+  Use TLS encryption.
+
+  .. code-block:: none
+
+    set service webproxy authentication ldap use-ssl
+
+
+.. cfgcmd:: set service webproxy authentication ldap username-attribute <attr>
+
+  Specifies the name of the DN attribute that contains the username/login.
+  Combined with the base DN to construct the users DN when no search filter is
+  specified (`filter-expression`).
+
+  Defaults to 'uid'
+
+  .. note:: This can only be done if all your users are located directly under
+    the same position in the LDAP tree and the login name is used for naming
+    each user object. If your LDAP tree does not match these criterias or if you
+    want to filter who are valid users then you need to use a search filter to
+    search for your users DN (`filter-expression`).
+
+  .. code-block:: none
+
+    set service webproxy authentication ldap username-attribute uid
+
+.. cfgcmd:: set service webproxy authentication ldap version <2 | 3>
+
+  LDAP protocol version. Defaults to 3 if not specified.
+
+  .. code-block:: none
+
+    set service webproxy authentication ldap version 2
+
+URL filtering
+=============
+
+.. include:: /_include/need_improvement.txt
+
+
+.. cfgcmd:: set service webproxy url-filtering disable
+
+  Disables web filtering without discarding configuration.
+
+  .. code-block:: none
+
+    set service webproxy url-filtering disable
+
+*********
+Operation
+*********
+
+.. include:: /_include/need_improvement.txt
+
+Filtering
+=========
+
+Update
+------
+
+If you want to use existing blacklists you have to create/download a database
+first. Otherwise you will not be able to commit the config changes.
+
+
+.. opcmd:: update webproxy blacklists
+
+  Download/Update complete blacklist
+
+  .. code-block:: none
+
+    vyos@vyos:~$ update webproxy blacklists
+    Warning: No url-filtering blacklist installed
+    Would you like to download a default blacklist? [confirm][y]
+    Connecting to ftp.univ-tlse1.fr (193.49.48.249:21)
+    blacklists.gz        100% |*************************************************************************************************************| 17.0M  0:00:00 ETA
+    Uncompressing blacklist...
+    Checking permissions...
+    Skip link for   [ads] -> [publicite]
+    Building DB for [adult/domains] - 2467177 entries
+    Building DB for [adult/urls] - 67798 entries
+    Skip link for   [aggressive] -> [agressif]
+    Building DB for [agressif/domains] - 348 entries
+    Building DB for [agressif/urls] - 36 entries
+    Building DB for [arjel/domains] - 69 entries
+    ...
+
+    Building DB for [webmail/domains] - 374 entries
+    Building DB for [webmail/urls] - 9 entries
+
+    The webproxy daemon must be restarted
+    Would you like to restart it now? [confirm][y]
+
+    [ ok ] Restarting squid (via systemctl): squid.service.
+    vyos@vyos:~$
+
+.. opcmd:: update webproxy blacklists category <category>
+
+  Download/Update partial blacklist.
+
+  Use tab completion to get a list of categories.
+
+* To auto update the blacklist files
+
+  :code:`set service webproxy url-filtering squidguard auto-update
+  update-hour 23`
+
+* To configure blocking add the following to the configuration
+
+  :code:`set service webproxy url-filtering squidguard block-category ads`
+
+  :code:`set service webproxy url-filtering squidguard block-category malware`
 
 Bypassing the webproxy
-^^^^^^^^^^^^^^^^^^^^^^
+----------------------
+
+.. include:: /_include/need_improvement.txt
 
 Some services don't work correctly when being handled via a web proxy.
 So sometimes it is useful to bypass a transparent proxy:
@@ -153,5 +403,32 @@ So sometimes it is useful to bypass a transparent proxy:
   (This can be useful when a called service has many and/or often changing
   destination addresses - e.g. Netflix.)
 
-.. _Squid3: http://www.squid-cache.org/
-.. _Squidguard: http://www.squidguard.org/
+********
+Examples
+********
+
+.. code-block:: none
+
+  vyos@vyos# show service webproxy
+   authentication {
+       children 5
+       credentials-ttl 60
+       ldap {
+           base-dn DC=example,DC=local
+           bind-dn CN=proxyuser,CN=Users,DC=example,DC=local
+           filter-expression (cn=%s)
+           password Qwert1234
+           server ldap.example.local
+           username-attribute cn
+       }
+       method ldap
+       realm "VyOS Webproxy"
+   }
+   cache-size 100
+   default-port 3128
+   listen-address 192.168.188.103 {
+       disable-transparent
+   }
+
+.. _Squid: http://www.squid-cache.org/
+.. _SquidGuard: http://www.squidguard.org/