wiki:StatsModule

Version 6 (modified by naokikambe, 10 years ago) (diff)

some minor changes

Document: How stats module collects data

  1. Introduction
  2. About stats module
  3. Procedure of stats module
    1. Basic procedure
    2. Startup procedure
    3. Collecting procedure
    4. Reporting procedure
    5. Config change procedure
    6. Shutdown procedure
  4. Collecting items
  5. Reporting items
  6. Available commands in bindctl
  7. Stats channel
  8. Backend DB for stats module
  9. Spec files
    1. Spec file for stats module
    2. Spec file for boss module
    3. Spec file for auth module
  10. In future releases

Introduction

This document is described about how stats module collect statistics data from other modules in BIND 10. Other modules are only Boss and Auth in initial version. This document is valid for initial version of stats module.

About stats module

  • Stats module collects stats data from each module.
  • Stats module reports statistics information via bindctl.
  • Stats module and other modules communicate by CC protocol.
  • Stats module is waiting for coming data from other modules, then other modules are sending data to stats module periodically.
  • Other modules send stats data to stats module independently from logic of stats module, so the frequency of sending data may not be constant.
  • Stats module collects data and aggregates it.
    +------ Sending modules -----+
    | +------+ +------+ +------+ |
    | | Boss | | Auth | | etc. | | <- *1
    | +------+ +------+ +------+ |
    +-----^--------^--------^----+
          |        |        |
          +--[CC protocol]--+
                   |               <- *2
                   v
            +--------------+  
            |    Stats     |       <- *3
            +--------------+ 
                   |               <- *4
                   v 
          +-----------------+
          |    Cmd-ctrld    |      <- *5
          +-----------------+
    
    *1 In initial version, statistics data of boss module and auth module
       are collected. But in future release, statistics data of other
       modules may be also collected, which are like Xfrin module, Xfrout
       module, CC-protocol module and so on.
    *2 Each module periodically send stats data to stats module via CC
       protocol.
    *3 Stats module collects data and aggregate it.
    *4 Stats module reports stats message with aggregated data.
    *5 Commands related to statistics are invoked by BIND 10 user.
    

Procedure of stats module

Basic procedure

Startup procedure

  1. Boss starts stats daemon and other modules.
  2. Spawned stats module contacts config manager.
  3. Stats starts to subscribe in stats channel.
              spawns  
    +------+ -------> +-------+ create ConfigManager
    | Boss |    	  | Stats | instance
    +------+ <------- +-------+
                ack
    

Collecting procedure

  1. Other modules send stats data to stats module periodically.
  2. Stats module collects it and then aggregates.
            
    +------+  sends
    | Boss | -------> +-------+ collecting
    +------+  	  | Stats |  / aggregating
    +------+ -------> +-------+
    | Auth |  sends
    +------+
    

Reporting procedure

  1. When commands related stats module like "show", are invoked via bindctl, stats daemon reports a result like formatted statistics data via bindctl.
            invoke
            command                send command
             ------> +-----------+ -----------> +-------+
    [ User ]         | Cmd-Ctrld |              | Stats |
             <------ +-----------+ <----------- +-------+
            print                  send result
            to stdout         (aggregated and formatted)
    

Config change procedure

  1. Configuration changes and commands from cmd-ctrld are come from config manager.
                   set_config		      config_update
    +-----------+ -----------> +---------------+ -----------> +-------+
    | Cmd-ctrld |  		   | ConfigManager |   		  | Stats |
    +-----------+ <----------- +---------------+ <----------- +-------+
    	          ack   		      	  ack
    

Shutdown procedure

  1. When Boss is shutting down or "shutdown" command for stats module is invoked, stats module is killed.
           kills
+------+ -------> +-------+
| Boss |    	  | Stats |
+------+ <------- +-------+
            ack

Collecting items

Stats module collects following items from boss and auth.

  • For Boss module
    • timestamp -- Milli-seconds of current time since epoch time (1970-01-01T00:00:00Z)
    • sequence -- A sequence number which must be unique and consistent in the sending module
    • boot_time -- A date time when BIND 10 starts up, format is YYYY-MM-DDTHH:MM:SSZ
  • For Auth module
    • timestamp -- Milli-seconds of current time since epoch time (1970-01-01T00:00:00Z)
    • sequence -- A sequence number which must be unique and consistent in the sending module
    • tcp -- A number of query counts per a process which Auth servers receives in TCP since it sends last
    • udp -- A number of query counts per a process which Auth servers receives in UDP since it sends last

Reporting items

Stats module reports following items via bindctl.

  • Report time -- A date time when stats module reports
  • bind10.LocalName -- A localname of boss given via CC protocol
  • bind10.BootTime -- A date time when BIND 10 process starts
  • auth.LocalName -- A localname of auth given via CC protocol
  • auth.queries.tcp -- A calculated query counts in TCP by stats module
  • auth.queries.udp -- A calculated query counts in UDP by stats module

Output image of stats data via bindctl:

* BIND 10 Statistics report
  Report time: ...

  bind10.LocalName: xxxx@localhost
  bind10.BootTime: ...

  auth.LocalName: xxx@ahost
  auth.queries.tcp: ...
  auth.queries.udp: ...

Available commands in bindctl

Following commands via bindctl are available in initial version of stats module.

"show" command
Stats module aggregates current numbers and show the list of them by using formatted text. specify the module name in the argument.
"clear" command
Stats module resets query counts to zero. If this command is invoked, then at first 'Are you sure?' prompt to confirm it.
"shutdown" command
shutdown command of Stats Module
"help" command
help command for stats module

Examples of command line:

> Stats help
Module  Stats    Statistics Module
Available commands:
         help		(Get help for module)
         show [module_name] (show statistics data)
         clear [module_name] (clear statistics data)
         shutdown       (Shut down stats module)
> Stats show
* BIND 10 Statistics report for all modules
  Report time: ...

  bind10.LocalName: xxxx@localhost
  bind10.BootTime: ...

  auth.LocalName: xxx@ahost
  auth.queries.tcp: ...
  auth.queries.udp: ...
> Stats show Boss
* BIND 10 Statistics report for Boss module 
  Report time: ...

  bind10.LocalName: xxxx@localhost
  bind10.BootTime: ...
> Stats show Auth
* BIND 10 Statistics report for Auth module
  Report time: ...

  auth.LocalName: xxx@ahost
  auth.queries.tcp: ...
  auth.queries.udp: ...
> Stats clear
Are you sure? :yes
statistics data for all modules cleared.
>

Stats channel

stats channel to send/subscribe stats data:

Port number
in CC protocol(msgq)
9912
(same number in current version)
Group name
to send/subscribe in CC protocol
Stats
(maybe case sensitive)

Backend DB for stats module

Stats module in initial version requires no specific Backend DB, such as sqlite3 or Berkeley DB. Stats module may store data only in python variables in memory. It may depend on python runtime.

Spec files

Followings are spec files for Stats module, Boss module, and Auth module extended from current version. Addtional items are followings. These items also have meaning of statistics data schema for each module, so they are expressed by JSON schema.

  • Add "stats_spec" object in "module_spec" object, which is corresponding to collecting items of each module.
  • Add "stats_output_spec" in "config_data" object, which is corresponding to reporting items of stats module.

Spec file for stats module

stats.spec:

{
  "module_spec": {
    "module_name": "Stats",
    "module_description" "Statistics Module",
    "config_data": [ // reporting items for stats module
      {
        "item_name": "stats_output_spec",
        "item_type": "array",
        "item_optional": false,
        "item_default": [
          {
            "name": "report_time",
            "title": "Report time",
            "description": "A date time when stats module reports",
            "type": "string",
            "format": "date-time"
          },
          {
            "name": "bind10.local_name",
            "title": "bind10.LocalName",
            "description": "A localname of Boss given via CC protocol",
            "type": "string"
          },
          {
            "name": "bind10.boot_time",
            "title": "bind10.BootTime",
            "description": "A date time when BIND 10 process starts",
            "type": "string",
            "format": "date-time"
          },
          {
            "name": "auth.local_name",
            "title": "auth.LocalName",
            "description": "A localname of Auth given via CC protocol",
            "type": "string",
            "format": "date-time"
          },
          {
            "name": "auth.queries.tcp",
            "title": "auth.queries.tcp",
            "description": "A number of incremental query counts per a process which Auth servers receives in TCP since last sending",
            "type": "number"
          },
          {
            "name": "auth.queries.udp",
            "title": "auth.queries.udp",
            "description": "A number of incremental query counts per a process which Auth servers receives in UDP since last sending",
            "type": "number"
          }
        ]
      }
    ],
    "commands": [ // commands related to stats module
      {
        "command_name": "show",
        "command_description": "Show statistics data",
        "command_args": [
          {
            "item_name": "module_name",
            "item_type": "string",
            "item_optional": true,
            "item_default": null
          }
        ]
      },
      {
        "command_name": "clear",
        "command_description": "Clear statistics data",
        "command_args": [
          {
            "item_name": "module_name",
            "item_type": "string",
            "item_optional": true,
            "item_default": null
          }
        ],
        "command_flag": "dangerous" // for "Are you sure?" prompt (This is just a idea.)
      },
      {
        "command_name": "shutdown",
        "command_description": "Shut down Stats module",
        "command_args": []
      }
    ],
    "stats_spec": [] // disable for stats module itself
  }
}

Spec file for boss module

bob.spec:

{
  "module_spec": {
    "module_name": "Boss",
    "module_description": "Master process",
    "config_data": [
    ],
    "commands": [
      {
        "command_name": "shutdown",
        "command_description": "Shut down BIND 10",
        "command_args": []
      }
    ],
    "stats_spec": [ // collecting items for boss module
      {
        "name": "bind10.timestamp",
        "description": "Milli-seconds of current time since epoch time (1970-01-01T00:00:00Z)",
        "type": "number",
        "format": "utc-millisec",
      },
      {
        "name": "bind10.sequence",
        "description": "A sequence number which must be unique and consistent in the sending module"
        "type": "number"
      },
      {
        "name": "bind10.boot_time",
        "description": "A date time when BIND 10 starts up, format is YYYY-MM-DDTHH:MM:SSZ",
        "type": "string",
        "format": "date-time"
      }
    ]
  }
}

Spec file for auth module

auth.spec:

{
  "module_spec": {
    "module_name": "Auth",
    "module_description": "Authoritative service",
    "config_data": [
      { "item_name": "database_file",
        "item_type": "string",
        "item_optional": True,
        "item_default": "/home/kambe/bind10/var/bind10-devel/zone.sqlite3"
      }
    ],
    "commands": [
      {
        "command_name": "shutdown",
        "command_description": "Shut down authoritative DNS server",
        "command_args": []
      }
    ],
    "stats_spec": [ // collecting items for auth module
      {
        "name": "auth.timestamp":
        "description": "Milli-seconds of current time since epoch time(1970-01-01T00:00:00Z)",
        "type": "number",
        "format": "utc-millisec"
      },
      {
        "name": "auth.sequence",
        "description": "A sequence number which must be unique and consistent in the sending module",
        "type": "number"
      },
      {
        "name": "auth.queries.tcp",
        "description": "A number of query counts per a process which Auth servers receives in TCP since it sends last",
        "type": "number"
      },
      {
        "name": "auth.queries.udp",
        "description": "A number of query counts per a process which Auth servers receives in UDP since it sends last",
        "type": "number"
      }
    ]
  }
}

In future releases

  • Will support output in XML format
  • Will add more collecting and reporting items

Attachments (6)

Download all attachments as: .zip