Changes between Version 50 and Version 51 of ControlAPIDesign


Ignore:
Timestamp:
Aug 18, 2018, 12:54:49 AM (15 months ago)
Author:
vicky
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • ControlAPIDesign

    v50 v51  
    11= Command Control Channel Design =
    2 == Introduction ==
    3 This document describes a design of the Control API in Kea. This API is used to manage Kea servers' configuration, control the state of the servers (e.g. shutdown or startup a server) and retrieve information about servers' states (e.g. how many leases have been assigned). The API allows for secure remote access to the Kea servers, i.e. the encryption and authentication mechanisms are used if required by the Kea configuration.
    4 
    5 This design is intended to satisfy the requirements listed in the [[wiki:ControlAPIRequirements|Requirements Document]].
    6 
    7 '''This is ongoing work and the document is a subject to updates.'''
    8 
    9 Sections marked as ''Historical'' are not a part of the actual design. They are held in this document as a reference to past design discussions, which resulted in choosing alternative solutions.
    10 
    11 == System Overview ==
    12 [[Image(control-api-system-diagram-proxy.png)]]
    13 
    14 The component diagram includes two systems: System A and System B. The former runs the Kea services, the latter runs a Control API Client (CAC). It is possible to run CAC on the same machine as Kea services, but in the typical case they will be installed on separate machines. Kea comprises three services: DHCPv4, DHCPv6 and D2. They are already shipped in the Kea package. DHCPv4 and DHCPv6 services can be controlled via commands sent over unix domain sockets. This is currently not supported for the D2 service, but is planned to be supported in one of the future releases.
    15 
    16 The design introduces new component, the Kea Control Agent (CA). It aggregates commands sent from the remote clients and forwards them to the appropriate Kea services.  CA runs HTTP service on the CAC facing link. A HTTP request received from the CAC encapsulate JSON-formatted commands within the HTTP payload. The CA strips the HTTP headers and retrieves the command. It then retrieves a command name from the JSON structure, as well as the name/class of a service which the command pertains to. Most of the commands are simply forwarded to one of the services. However, some commands should be handled by the CA on its own. A common example of the command that is not forwarded is a "startup" command, which instructs the CA to launch a particular Kea service.
    17 
    18 ''Note: "startup" command is not explicitly listed as a requirement for Kea 1.2.0 release, but it serves as a good example of a command to be handled by CA. This command will be implemented in one of the future releases.''
    19 
    20 When the CA receives a response from the service to which the command was forwarded, it creates a HTTP response by encapsulating the received JSON response with HTTP headers. If the command is handled by the CA on its own, the CA is responsible for creating a JSON response.
    21 
    22 The third party HTTP server ("nginx" on our diagram) serves as a '''reverse proxy''' which forwards requests to the CA. The primary role of the proxy server is to provide a secure communication channel (HTTP + TLS) between CA running on System A, and the CAC on System B. The CA itself doesn't provide any encryption and authentication mechanisms. The communication between the proxy and the CA is unencrypted and not authenticated.
    23 
    24 It is not necessary to install proxy server when securing the communication is not required. The CA can be configured to expose the HTTP service to the CAC directly. However, because the communication is not secured, any malicious client can take control over the Kea services. A possible way to mitigate this problem is to setup firewall rules to restrict the group of hosts that can communicate with the webservice.
    25 
    26 == Extended JSON Command Structure ==
    27 In most cases, when the CA receives a command from the CAC, it will forward this command to one of the Kea servers for processing. When the response from the server is received, CA will simply send it encapsulated in the HTTP response to the CAC. Command names aren't unique between the Kea services. Thus, the CA needs some additional information from the CAC to forward the command to the appropriate service. In some cases commands are not forwarded but they are handled by the CA on its own. An example is the "startup" command which instructs the CA to start a specified Kea service, e.g. DHCPv4 server.
    28 
    29 ''Note: "service-startup" is not explicitly listed in the requirements for the Kea 1.2.0 release.''
    30 
    31 Regardless  if the command is forwarded or handled by the CA directly, the CAC must specify the service (or class of the services) which the command pertains to. The exceptions are these commands that do not trigger actions on the particular Kea services, e.g. commands that are specific to CA (e.g. reload CA's configuration) or some  global administrative commands. None of those are currently specified, but can be added in the future.
    32 
    33 An example of a command which specifies the service name is:
    34 {{{
    35 {
    36     "service": "dhcp4",
    37     "command": "foo",
    38     "arguments": {
    39         "param_foo": "value1",
    40         "param_bar": "value2",
    41         ...
    42     }
    43 }
    44 }}}
    45 
    46 In the example above, the command "foo" pertains to the DHCPv4 service. Other possible values are "dhcp6" or "d2".
    47 
    48 In the example above the "param_foo" and "param_bar" are simple strings, but in general they may contain much more complex structures. Consider ''add-subnet4'' command which adds a new subnet to the configuration. The single parameter "subnet" will contain a map describing the new subnet just as it looks in the Kea configuration file.
    49 
    50 Currently we're assuming that a single instance of the particular Kea service is being controlled by the CA. It is envisaged that in the future CA will be able to manage multiple services of the same type running on the particular machine, e.g. multiple DHCPv4 servers. In such case, the "service" parameter will specify that the command pertains to all managed instances of a given service. Such command will be forwarded to all instances and the received responses are combined within a single JSON list. For example:
    51 
    52 {{{
    53 [
    54     {
    55         "pid": 8080,
    56         "result": 0,
    57         "error": "All ok"
    58     },
    59     {
    60         "pid": 8081,
    61         "result": 0,
    62         "error": "All ok"
    63     }
    64 ]
    65 }}}
    66 
    67 Note that "pid" values are added to the responses to identify the process which responded to the particular command.
    68 
    69 As already mentioned, in the Kea 1.2.0, the CA can only manage a single instance of the particular service. Thus, the response list will contain at most one result, e.g.:
    70 
    71 {{{
    72 [
    73     {
    74         "pid": 8080,
    75         "result": 0,
    76         "error": "All ok"
    77     }
    78 ]
    79 }}}
    80 
    81 The following part:
    82 {{{
    83     {
    84         "pid": 8080,
    85         "result": 0,
    86         "error": "All ok"
    87     }
    88 }}}
    89 
    90 The square brackets [ ]  are added by the CA.
    91 
    92 In the future, any command that pertains to a single instance of a particular service will contain a "pid" parameter which holds a process id of this instance, instead of "service" parameter. This parameter, however, is not to be supported in the Kea 1.2.0 release.
    93 
    94 == Restructuring !CommandMgr ==
    95 !CommandMgr is a singleton object currently serving three purposes:
    96 - maintain associations of supported commands with the appropriate handler functions,
    97 - open and maintain the connection (via unix domain socket) with control API clients,
    98 - report supported commands by responding to 'list-commands' command.
    99 
    100 With the advent of Kea 1.2.0 release the command handlers can be implemented both in the core Kea code and optional hook libraries. The new component, CA, will also use hooks mechanism to provide external implementations of some of the commands.
    101 
    102 Neither CA nor hook libraries attached to the Kea services require UNIX domain sockets functionality of the !CommandMgr, but they do require ability to maintain the mapping of commands to their handlers. Thus, the functions of the !CommandMgr must be decoupled as presented on the figure below.
    103 
    104 [[Image(command-mgr-decouple.svg)]]
    105 
    106 The !CommandMgrBase provides methods for registering, deregistering command handlers. The CommandMgrBase::processCommand finds appropriate handlers for a processed commands and executes them.
    107 
    108 The !CommandMgrHooked extends CommandMgr::processCommand with new hook points 'control_command_receive' and 'control_commands_list'.
    109 
    110 The !CommandMgr derives from !CommandMgrHooked and implements mechanisms responsible for communication via unix sockets.
    111 
    112 == New Hook Points ==
    113 It is desired that Kea servers and Control Agent can handle additional commands implemented within hook libraries. These commands may be specific to a particular deployment and thus can't be natively supported by Kea. The new hook point 'control_command_receive' will be added within the CommandMgrHooked::processCommand and will be common  for all applications and libraries using/deriving from this class. The callouts for this hook point will be executed when the command is received by the process and before the process handles the command on its own. If a hook library handles the command on its own, it should return a 'next action' flag set to 'skip'. The CommandMgrHooked::processCommand will assume that the command has been handled and will return the response to the client. If the 'skip' status is not set, the !CommandMgrHooked::processCommand will handle the command if the appropriate handler has been registered.
    114 
    115 The 'control_command_receive' callouts will be executed with the following arguments:
    116 - name: ''command'', type: !ConstElementPtr, direction: '''in'''
    117 - name: ''response'', type: !ElementPtr, direction: '''out'''
    118 
    119 The ''list-commands'' is a special command which returns commands supported by the module. If the server receiving this command has hook libraries implementing additional commands attached, the returned list should combine the natively supported commands and these supported by the hook libraries. The !CommandMgrHooked::processCommand adds a special hook point 'control_commands_list' which should be implemented by the hook libraries providing new commands. The callouts for this hook point will be executed with the following argument:
    120 - name: ''commands_list'', type: !ElementPtr, direction: ''out''
    121 
    122 The ''commands_list'' is a pointer to the JSON list of commands supported by the hook library. The hook library should insert the list of commands into this structure.
    123 
    124 The !CommandMgrHooked::processCommand "glues" commands returned by the hook libraries with the natively supported commands and returned them to the control client.
    125 
    126 Hook libraries should use !CommandMgrBase as a base class for implementing their own instances of Command Manager, i.e. each hook library has its own Command Manager instance used to manage commands supported by this library.
    127 
    128 == Handling Commands within Control Agent ==
    129 The Kea Control Agent (CA) is a process which receives commands from the clients and either forwards them to the respective Kea services or handles them on its own. CA determines if it should handle the command on its own by checking if the command handler has been registered for it.
    130 
    131 When the CA receives a command, it first executes 'control_command_receive' callouts for this command. If the callouts return 'skip' status it simply returns the response to the CAC. Otherwise, the CA checks if it should handle the command on its own or forward to one of the Kea servers. This is done by checking if the command handler for this command has been registered. If it has been registered, this handler is executed to perform the command and generate response. If there is no handler for this command registered, the command is forwarded to the service specified as "service" parameter within JSON command.
    132 
    133 == Control Agent ==
    134 This section presents the design of the Control Agent.
    135 
    136 === Configuration ===
    137 The CA shares configuration format with all other Kea services. The CA can either use the dedicated configuration file or simply read the CA specific configuration from the configuration file common for all services. The following is a basic configuration of the CA.
    138 
    139 {{{
    140 "Control-agent":
    141 {
    142     "http-host": "localhost",
    143     "http-port": 8000,
    144     "control-sockets":
    145     {
    146         "dhcp4-server":
    147         {
    148             "socket-type": "unix",
    149             "socket-name": "/path/to/the/unix/socket-v4"
    150         },
    151 
    152         "dhcp6-server":
    153         {
    154             "socket-type": "unix",
    155             "socket-name": "/path/to/the/unix/socket-v6"
    156         },
    157 
    158         "d2-server":
    159         {
    160             "socket-type": "unix",
    161             "socket-name": "/path/to/the/unix/socket-d2"
    162         }
    163     },
    164 
    165    "hooks-libraries": [
    166    {
    167         "library": "/opt/local/control-agent-commands.so",
    168         "parameters": {
    169             "param1": "foo"
    170          }
    171     }
    172  ]
    173 
    174 },
    175 
    176 "Logging":
    177 {
    178   "loggers": [
    179     {
    180       "name": "kea-ctrl-agent",
    181       "output_options": [
    182           {
    183             "output": "/Users/marcin/devel/kea-build/var/log/kea-ctrl-agent.log"
    184           }
    185       ],
    186       "severity": "INFO",
    187       "debuglevel": 0
    188     }
    189   ]
    190 }
    191 }}}
    192 
    193 The configuration example provided above specifies the host and port on which the HTTP service is available. It also specifies unix domain sockets which are used for communication with the respective Kea servers. If the socket configuration for the particular service is not specified, the control socket for this service is not opened. The "socket-type": "unix" is currently the only value supported by the CA.
    194 
    195 === Debate About Architecture ===
    196 One of the major problems to be resolved by the design was to determine how to implement reliable and secure HTTP service, and avoid spending too much engineering time on creation and maintenance of the proprietary code. The aspects that dominated the discussion between the team members were:
    197 - Kea is written in C++. Can Control Agent be implemented in other language, with a better support for creating web services?
    198 - Should Kea crypto library be extended to provide a secure HTTP connection (over TLS) or we rather want to rely on third party code to solve the issue of authentication and encyrption?
    199 - If third party code is used, is this ok that it comes with dependencies on OpenSSL?
    200 - Should Control Agent support HTTP/1.1 or HTTP/2?
    201 - Should Control API provide authentication via HTTP BA (Basic Authentication)?
    202 
    203 Starting from the least contentious aspects the team has decided that:
    204 - HTTP BA would be useful if Control API provided multiple administration roles. Given there is only one administration role, the authentication problem can be resolved by using TLS certificates installed on the limited number of hosts which are authorized to use the API.
    205 - HTTP/2 seems to be much more complex than HTTP/1.1 and not fully deployed, so we rather want to use HTTP/1.1. Because there is backward compatibility between version 1.1 and 2.0, the 2.0 can be implemented later if we desire.
    206 - Kea Control Agent can be implemented in another language (not C++) if the benefits outweigh the issues with additional complexity related to multiple languages and more dependencies.
    207 
    208 Before the team was able to conclude what other language would best work for this the new aspects were considered, which implied that C++ can actually be a better choice than anything else. They are:
    209 - Kea already provides a lot of useful code for the Control Agent, e.g. CommandMgr, logging, configuration parsing (currently transitioned to use Bison),
    210 - If the security layer can be provided by the use of HTTP proxy server (nginx, Apache or other) the most complex part (security) is already solved elsewhere. This leaves us with only a problem of providing a minimal HTTP server implementation on the Kea side.
    211 - There are some example solutions available online for the HTTP service implementation in C++. One of them is [[http://www.boost.org/doc/libs/1_55_0/doc/html/boost_asio/examples/cpp11_examples.html|boost::asio based solution]], another one is the FastCGI based solution.
    212 
    213 The proof of concepts for both solutions have been created and put into the internal Kea repository on the following branches:
    214 - experiments/http
    215 - experiments/fcgi
    216 
    217 ''Note that only ISC employees currently have access to these branches''.
    218 
    219 In the opinion of the author of this design the boost::asio based solution is better because it doesn't introduce new dependencies (FastCGI not required). In addition, it is asynchronous solution relying on asynchronous reads/writes, making it easier to multiplex between multiple connections (HTTP, multiple Unix sockets) and support asynchronous events, such as interval timers etc.
    220 
    221 Finally, the team concluded that we should not attempt to implement TLS in Kea crypto library because both OpenSSL and Botan have awkward API which, if misused, can introduce security vulnerabilities in Kea. Therefore, we agreed that we should rely on the third party software (e.g. nginx or Apache etc) to provide a reverse proxy function. The major advantage of this choice is that we offload responsibility for security layer to third party. We also allow the users to select a HTTP server of their liking  to be used as a proxy.
    222 
    223 === Proposed Architecture ===
    224 The simplified class diagram of the Control Agent service implemented using boost::asio (see previous section) is depicted below.
    225 
    226 [[Image(libkea-http-class-diagram.svg)]]
    227 
    228 The HTTP service is built around the !HttpListener object which creates a TCP socket and accepts new connections on this socket. Once the connection comes in, the !HttpConnection object is created to represent this new connection. The !HttpConnectionPool serves as a container for all '''active''' connections. If the CA process has to shut down, all connections from this pool are closed before it shuts down. Once the HTTP transaction is completed, the connection used for this transaction is also closed and removed from the pool.
    229 
    230 !HttpConnection is used to read and parse HTTP requests received. !HttpConnection creates an instance of the !HttpRequestParser which implements a state machine used for parsing the HTTP requests. Each connection comes with its own instance of the parser. The parser stores an instance of the derivation of the !HttpRequest object, which is used to interpret the received request once it has been parsed. The Control Agent uses !PostHttpRequestJson class as a request interpreter as it provides means for interpreting JSON structures encapsulated within HTTP. It also checks that the received message is HTTP POST and that the "Content-Type" is "application/json".
    231 
    232 The interpreted requests are held within the !HttpRequest objects and are used to generate !HttpResponse, which represent responses to be sent to the CAC. The !HttpRequestCreator class provide basic tool set for generating HTTP responses. The derived classes provide specialization of this class to generate the body for these responses. For example, the !CtrlAgentResponseCreator, being a part of the CA application will generate the !HttpResponses containing JSON body representing answers to the commands issued by CAC. This class will obviously use a derivation of the !CommandMgrBase to map JSON queries to command handlers.
    233 
    234 The HTTP specific classes will be placed in a new library: libkea-http so as they can be reused in other components in the future. These classes are marked in yellow on the diagram above. The classes external to the new HTTP library are marked in blue.
    235 
    236 === HTTP Parser State Diagram ===
    237 The !HttpRequestParser is the complex part of the libkea-http library. Its state model is based on the implementation of the [[http://www.boost.org/doc/libs/1_55_0/doc/html/boost_asio/examples/cpp11_examples.html | sample HTTP server in boost]]. The following diagram presents a simplified state model of the HTTP parser.
    238 
    239 [[Image(libkea-http-parser-states.svg)]]
    240 
    241 Note that almost all states on this diagram can transition to "parse failure" state, but these transitions are not shown on this diagram to keep the diagram clear. The transition to "parse failure" occurs when unexpected character is found in the stream. For example, the parser received CR character and expects LF. If LF is not found, parsing fails.
    242 
    243 For better understanding of the transitions between the states we provide a sample HTTP message which is parsed using this model.
    244 
    245 {{{
    246 POST /foo/bar HTTP/1.0\r\n
    247 Content-Type: application/json\r\n"
    248 Content-Length: 45\r\n
    249 User-Agent: Kea/1.2 Command \r\n"
    250  Control \r\n"
    251 \tClient\r\n\r\n"
    252 { "service": "dhcp4", "command": "shutdown" }
    253 }}}
    254 
    255 The first line contains HTTP method, URI and version number. This is followed by 3 HTTP headers: "Content-Type", "Content-Length" and "User-Agent". The "User-Agent" has line breaks marked by Leading White Space (LWS) characters (space and tab). The HTTP message body contains command using JSON notation.
    256 
    257 == Documenting Supported Commands ==
    258 We have discussed whether the Command API could use .spec (JSON) files to define supported commands. We considered whether the .spec files could be used as input for generating documentation as well as for configuring !CommandMgr. We concluded that it brings a lot of complexity and puts additional burden on the implementors of the hook libraries with a minimal gain. Thus, the current design excludes this feature from the Kea 1.2.0 release.
    259 
    260 Nevertheless, the format of the .spec file is convenient to document the commands we're planning to support within this design. Hence, we paste it here to demonstrate the basic commands we're aiming to support in Kea 1.2.0.
    261 
    262 {{{
    263 {
    264     "module": "dhcp4-server",
    265     "commands": [
    266          {
    267              "command": "shutdown",
    268              "since_version": "0.9.2",
    269              "in_subversion": "",
    270              "description": "Terminates the server process.",
    271              "arguments": [ ]
    272          },
    273          {
    274              "command": "reload-config",
    275              "since_version": "1.0.0",
    276              "in_subversion": "",
    277              "description": "Instructs the server to reload configuration from a file.",
    278              "arguments": [
    279                  {
    280                      "argument": "location",
    281                      "type": "text",
    282                      "mandatory": false
    283                  }
    284              ]
    285          },
    286          {
    287              "command": "get-config",
    288              "since_version": "1.2.0",
    289              "in_subversion": "",
    290              "description": "Retrieves entire server configuration.",
    291              "arguments": [ ]
    292          },
    293          {
    294              "command": "set-config",
    295              "since_version": "1.2.0",
    296              "in_subversion": "",
    297              "description": "Instructs the server to use new configuration. The configuration is not persisted to disk until write-config is invoked.",
    298              "arguments": [
    299                  {
    300                      "argument": "config",
    301                      "type": "text",
    302                      "mandatory": true
    303                  }
    304              ]
    305          },
    306          {
    307              "command": "test-config",
    308              "since_version": "1.2.0",
    309              "in_subversion": "",
    310              "description": "Performs dry run of the server configuration. It is used to test that the new configuration is valid.",
    311              "arguments": [
    312                  {
    313                      "argument": "config",
    314                      "type": "text",
    315                      "mandatory": true
    316                  }
    317              ]
    318          },
    319          {
    320              "command": "write-config",
    321              "since_version": "1.2.0",
    322              "in_subversion": "",
    323              "description": "Persists currently used configuration to disk.",
    324              "arguments": [
    325                  {
    326                      "argument": "location",
    327                      "type": "text",
    328                      "mandatory": true
    329                  }
    330              ]
    331          },
    332          {
    333              "command": "leases-reclaim",
    334              "since_version": "1.2.0",
    335              "in_subversion": "",
    336              "description": "Instructs the server to reclaim all expired leases.",
    337              "arguments": [ ]
    338          },
    339          {
    340              "command": "list-commands",
    341              "since_version": "1.0.0",
    342              "in_subversion": "",
    343              "description": "Returns all commands supported by the server and those that are planned to be supported in the future releases.",
    344              "arguments": [ ]
    345          }
    346     ]
    347 }
    348 }}}
    3492
    3503
    351 == Tasks ==
    352 The following is the list of implementation and documentation tasks required for the Control API feature in Kea 1.2.0:
    353 
    354 1. #5100 - Decouple !CommandMgr's unix sockets from commands handling and Implement 'control_command_receive' hook point.
    355 1. Implement basic commands (the actual ones to be implemented for 1.2.0 release is TBD)
    356 1. #5074 - Move DController and other generic classes to a separate library so as it can be reused by CA.
    357 1. #5075 - Create stub Control Agent
    358 1. #5076 - Implement configuration parsing within Control Agent (Bison)
    359 1. #3175 - d2::!StateModel class(es) to lib::util
    360 1. #5077 - Implement HTTP requests parser in libkea-http library
    361 1. #5088 - Create !HttpResponse and !HttpResponseCreator classes
    362 1. #5094 - Implement TCPAcceptor class in linkea-asiolink
    363 1. #5099 - Create !HttpListener, !HttpConnection and !HttpConnectionPool
    364 1. #5107 - Create !CtrlAgentResponseCreator class and link it with !CommandMgrBase
    365 1. #5078 - Control Agent: Create connections via unix sockets in Control Agent and forward received commands
    366 1. #5108 - Update keactrl to support kea-ctrl-agent startup, shutdown and reconfiguration.
    367 1. #5079 - Document the solution
    368 
    369 == Considerations Regarding Technology Selection for Control Agent (Historical) ==
    370 The Kea Control Agent (CA) implements a web service that Command API Clients (CAC) use to deliver commands to the controlled system. The team considered whether CA should be implemented in C++, to be consistent with the source code of other Kea modules, or it can be implemented in another language, which provides better framework for building web services and is better suited for writing web applications in general.  The team eventually decided that the advantages provided by other languages outweigh the desire to avoid additional dependencies and having a common code base for the Kea modules.
    371 
    372 The following table lists candidate technologies arbitrarily selected for the implementation of the CA. The assumptions which led to picking this set of technologies for comparison were:
    373 - Technology provides some framework for building web services with minimal effort,
    374 - Technology is known to be in widespread use, has good support and wide community,
    375 - ISC staff has some experience with the technology,
    376 - The programming language associated with the technology has a straightforward syntax,
    377 - The language supports object oriented programming.
    378 
    379 Note that Perl has been excluded from this list as it can be awkward to use with object oriented programming. Ruby on Rails has been excluded as it is known to have hard to understand syntax for the beginners and seems to be too heavy for the simple purpose of the CA.
    380 ||==||=Python=||=PHP=||=Java=||=Nodejs=||
    381 ||Availability||High||High||High||High (mostly via Nodejs repos or website)||
    382 ||Integration with HTTP servers||yes (via module)||yes (via module)||yes (Tomcat, Jetty...)||partial (via reverse proxy)||
    383 ||Built-in HTTP server||yes (server class)||yes (since version PHP5)||in packages (via Jersey, JavaEE)||yes (via Express framework)||
    384 ||Support for JSON||yes||yes (as of PHP 5.2)||in libraries||yes||
    385 ||Unix sockets||yes||yes||via external library||yes||
    386 ||Unit tests||yes (unittest framework)||phpunit (phpunit.de)||JUnit||Built-in assert or external Mocha framework||
    387 ||SQL support||MySQL via pyPI or mysql-dev. Postgres from postgres packages||yes||yes (JDBC)||via npm||
    388 ||Issues||What version to choose: 2.7 or 3.0+?||PHP was invented for pages rendering||Many dependencies, optional packages, complex IDE||Some packages may be available from external sources, or unavailable||
    389 ||Pros||ISC has some experience in Python||Mature language, high availability||Cross platform, many libraries and features||Clear syntax, easy installation, community, ISC has some expertise||
     4Migrated to Gitlab. Please see: https://gitlab.isc.org/isc-projects/kea/wikis/designs/control-api-design