Opened 7 years ago

Closed 5 years ago

#2245 closed defect (wontfix)

cache-enable true causes crash at startup

Reported by: jreed Owned by:
Priority: medium Milestone: Remaining BIND10 tickets
Component: Unclassified Version: bind10-old
Keywords: Cc:
CVSS Scoring: Parent Tickets:
Sensitive: no Defect Severity: N/A
Sub-Project: Core Feature Depending on Ticket:
Estimated Difficulty: 8 Add Hours to Ticket: 0
Total Hours: 0 Internal?: no

Description

Setting cache-enable true at run-time doesn't fail. Later (stop and then) start of bind10 fails.

A similar detailed example was posted to the bind10-users list; see https://lists.isc.org/pipermail/bind10-users/2012-September/000391.html

My own example failed with:

2012-09-06 15:22:12.758 FATAL [b10-auth.auth] AUTH_SERVER_FAILED server failed: Auto-detection of zones to cache is not yet implemented, supply cache-zones parameter
2012-09-06 15:22:12.758 DEBUG [b10-auth.cc] CC_DISCONNECT disconnecting from message queue daemon
2012-09-06 15:22:12.758 DEBUG [b10-auth.cc] CC_UNSUBSCRIBE unsubscribing from communication group data_sources
...
2012-09-06 15:22:13.811 INFO  [b10-boss.boss] BIND10_SEND_SIGTERM sending SIGTERM to msgq (PID 20075)
2012-09-06 15:22:13.812 [b10-msgq] Closing socket fd 9
[b10-msgq] Receive error: EOF
[b10-msgq] Closing socket fd 8
[b10-msgq] Receive error: EOF
[b10-msgq] Closing socket fd 5
[b10-msgq] Receive error: EOF
[b10-msgq] Closing socket fd 7
[b10-msgq] Receive error: EOF
INFO  [b10-boss.boss] BIND10_SEND_SIGTERM sending SIGTERM to b10-stats-httpd (PID 19437)
2012-09-06 15:22:13.822 INFO  [b10-boss.boss] BIND10_SEND_SIGTERM sending SIGTERM to b10-xfrout (PID 20911)
2012-09-06 15:22:13.829 INFO  [b10-boss.boss] BIND10_SEND_SIGTERM sending SIGTERM to b10-cmdctl (PID 7189)
2012-09-06 15:22:13.829 INFO  [b10-boss.boss] BIND10_SEND_SIGTERM sending SIGTERM to b10-stats (PID 28795)
Traceback (most recent call last):
  File "/home/reed/opt/bind10/libexec/bind10-devel/b10-xfrout", line 1065, in <module>
    xfrout_server = XfroutServer()
  File "/home/reed/opt/bind10/libexec/bind10-devel/b10-xfrout", line 936, in __init__
    self._cc.start()
  File "/home/reed/opt/bind10/lib/python3.1/site-packages/isc/config/ccsession.py", line 212, in start
    self.__send_spec()
  File "/home/reed/opt/bind10/lib/python3.1/site-packages/isc/config/ccsession.py", line 442, in __send_spec
    seq = self._session.group_sendmsg(msg, "ConfigManager")
  File "/home/reed/opt/bind10/lib/python3.1/site-packages/isc/cc/session.py", line 268, in group_sendmsg
    }, isc.cc.message.to_wire(msg))
  File "/home/reed/opt/bind10/lib/python3.1/site-packages/isc/cc/session.py", line 101, in sendmsg
    cc = self._socket.send(data)
socket.error: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/reed/opt/bind10/libexec/bind10-devel/b10-cmdctl", line 626, in <module>
    run(options.addr, options.port, options.idle_timeout, options.verbose)
  File "/home/reed/opt/bind10/libexec/bind10-devel/b10-cmdctl", line 585, in run
    CommandControl, idle_timeout, verbose)
  File "/home/reed/opt/bind10/libexec/bind10-devel/b10-cmdctl", line 483, in __init__
    self.cmdctl = CommandControlClass(self, verbose)
  File "/home/reed/opt/bind10/libexec/bind10-devel/b10-cmdctl", line 234, in __init__
    self._setup_session()
  File "/home/reed/opt/bind10/libexec/bind10-devel/b10-cmdctl", line 250, in _setup_session
    self.command_handler)
  File "/home/reed/opt/bind10/lib/python3.1/site-packages/isc/config/ccsession.py", line 198, in __init__
  File "/home/reed/opt/bind10/lib/python3.1/site-packages/isc/config/ccsession.py", line 419, in add_remote_config
    self._add_remote_config_internal(module_spec, config_update_callback)
  File "/home/reed/opt/bind10/lib/python3.1/site-packages/isc/config/ccsession.py", line 343, in _add_remote_config_internal
    answer, _ = self._session.group_recvmsg(False, seq)
  File "/home/reed/opt/bind10/lib/python3.1/site-packages/isc/cc/session.py", line 275, in group_recvmsg
    env, msg  = self.recvmsg(nonblock, seq)
  File "/home/reed/opt/bind10/lib/python3.1/site-packages/isc/cc/session.py", line 130, in recvmsg
    data = self._receive_full_buffer(nonblock)
  File "/home/reed/opt/bind10/lib/python3.1/site-packages/isc/cc/session.py", line 212, in _receive_full_buffer
    self._receive_len_data()
  File "/home/reed/opt/bind10/lib/python3.1/site-packages/isc/cc/session.py", line 172, in _receive_len_data
    new_data = self._receive_bytes(self._recv_len_size)
  File "/home/reed/opt/bind10/lib/python3.1/site-packages/isc/cc/session.py", line 160, in _receive_bytes
    raise ProtocolError("Read of 0 bytes: connection closed")
isc.cc.session.ProtocolError: Read of 0 bytes: connection closed
Exception socket.error: error(32, 'Broken pipe') in <bound method ModuleCCSession.__del__ of <isc.config.ccsession.ModuleCCSession object at 0x7f7ff53ed110>> ignored
2012-09-06 15:22:13.930 INFO  [b10-boss.boss] BIND10_PROCESS_ENDED process 7189 of b10-cmdctl ended with status 256
2012-09-06 15:22:13.931 INFO  [b10-boss.boss] BIND10_PROCESS_ENDED process 20911 of b10-xfrout ended with status 256
2012-09-06 15:22:13.931 INFO  [b10-boss.boss] BIND10_PROCESS_ENDED process 19437 of b10-stats-httpd ended with status 15
2012-09-06 15:22:13.931 INFO  [b10-boss.boss] BIND10_PROCESS_ENDED process 28795 of b10-stats ended with status 15
2012-09-06 15:22:13.931 INFO  [b10-boss.boss] BIND10_PROCESS_ENDED process 20075 of msgq ended with status 0
2012-09-06 15:22:13.932 INFO  [b10-boss.boss] BIND10_SHUTDOWN_COMPLETE all processes ended, shutdown complete

The msgq shutdown before users of it.

The second problem is that the error is fatal, but was no fatal when set via bindctl.

Third problem is that I can't run bindctl to fix it now.

Subtickets

Change History (8)

comment:1 follow-up: Changed 7 years ago by vorner

So, for which problem is this ticket?

First of, the value should have not been saved in this invalid version.

Second, I don't think that the order of SIGTERMs really matters. This is _emergency_ shutdown. Does it really matter if the processes crash when they would be killed right after that anyway? Sure, the order of SIGTERMs could be special-cased for msgq, but I don't see any problem except maybe little bit of aesthetic feeling.

comment:2 Changed 7 years ago by shane

The problem is that for a user it looks like everything fell apart, when the problem is specific and isolated. So I think the ticket is to kill the core applications last, especially msgq. :)

comment:3 Changed 7 years ago by vorner

I'm not sure the problem is isolated. In fact, what happened there was quite global (at least config manager, auth and a plugin across two different runs were involved to create this situation. I think the impression that everything fell apart is much closer to reality than pretending it is isolated.

Also, due to the state of the configuration framework minefield, I think this can happen quite everywhere. I don't know if we want to try to find a cure for each of the symptoms whenever they appear when we should do something about the configuration. But I know, no time :-|.

comment:4 Changed 7 years ago by shane

  • Milestone New Tasks deleted
  • Sub-Project changed from DNS to Core

comment:5 in reply to: ↑ 1 Changed 7 years ago by jreed

Replying to vorner:

So, for which problem is this ticket?

First of, the value should have not been saved in this invalid version.

Yes that is the problem of this ticket.

Another ticket #2249 is for the other issue.

comment:6 Changed 7 years ago by jreed

  • Summary changed from msgq stops too early (cache-enable true causes crash at startup) to cache-enable true causes crash at startup

comment:7 Changed 6 years ago by tomek

  • Milestone set to Remaining BIND10 tickets

comment:8 Changed 5 years ago by tomek

  • Resolution set to wontfix
  • Status changed from new to closed
  • Version set to old-bind10

This issue is related to bind10 code that is no longer part of Kea.

If you are interested in BIND10/Bundy framework or its DNS components,
please check http://bundy-dns.de.

Closing ticket.

Note: See TracTickets for help on using tickets.