Opened 2 years ago

Closed 17 months ago

#5555 closed enhancement (duplicate)

a receive queue controls.

Reported by: fdupont Owned by: fdupont
Priority: medium Milestone: Kea1.5
Component: Unclassified Version: git
Keywords: Cc:
CVSS Scoring: Parent Tickets:
Sensitive: no Defect Severity: N/A
Sub-Project: DHCP Feature Depending on Ticket:
Estimated Difficulty: 0 Add Hours to Ticket: 0
Total Hours: 0 Internal?: no


I propose two ways to control the receive queue to avoid big backlogs which can happen when servicing is too slow (and the fact clients retransmit of course does not help at all).

First is the POSIX setsockopt(SO_RCVBUF) which sets the maximum size in bytes of the socket receive queue. When the queue is full (i.e. an incoming packet is bigger than the maximum minus the current size) new packets are dropped instead added at the end of the queue.
This allows to avoid big backlog but as it drops new packets it is not the best/only solution.

Second idea is to use ioctl(FIONREAD) which returns the current size in bytes of the receive queue (very efficient system call BTW). I propose to use it in two ways:

  • when it returns a large value (threshold to determine) packets should be simply popped and dropped.
  • after servicing a packet it is more efficient to look at if there is another one than to come back to select (a real performance pig). Of course only a limited (another parameter to determine) number of packets should be serviced because the select loop includes other services.

About the last part of the second idea I refer to the AFTR code where I implemented this.
Note there is a big theoretical and practical background on the way to manage queue in high load / congestion situations, e.g RED (Random Early Detection). A good subject for a student...


Change History (11)

comment:1 Changed 2 years ago by fdupont

After some thinking it should be a good idea to import pending packets inside the application: packet (vs byte) sizing, better drop strategy, etc. Note this does not remove all interest in the setsockopt (i.e. enforce a reasonable kernel queue size), not in the ioctl (still better than go back to select() to read() packets if the read is blocking). And science can still be applied!

comment:2 Changed 2 years ago by tomek

  • Milestone changed from Kea-proposed to Kea1.5

As discussed during March 8th call, moving to 1.5.

comment:3 Changed 23 months ago by tomek

  • Milestone changed from Kea1.5 to Kea1.5 beta

Milestone renamed

comment:4 Changed 23 months ago by tomek

  • Milestone changed from Kea1.5 beta to Kea1.5

Milestone renamed

comment:5 Changed 20 months ago by fdupont

SO_RECVBUF is not the right solution:

  • when a queue is full all kernels drop the last incoming packet: we want the opposite.
  • DHCPv4 by default does not use an UDP socket.
  • LPF uses a raw socket so the setsockopt can apply but the default is likely different.
  • BPF is worst: packets are added into one buffer (fixed size, default 4096) and the code extracts only the first one.

In conclusion instead to limit queue size it is far better to just manage to have queues always empty or nearly empty.

comment:6 Changed 20 months ago by fdupont

  • Owner set to fdupont
  • Status changed from new to accepted

comment:7 Changed 20 months ago by fdupont

I ran some performance checks: with the ring/thread the throughput is a bit smaller and drops a bit higher. The real difference is in the service delay: it becomes very stable to a small value so IMHO the code performs exactly as expected.
Stalling the ticket waiting fro design review.

comment:8 Changed 20 months ago by fdupont

Things to do:

  • add a configurable size of the ring (today it is constant and is 5).
  • make the mechanism optional in the code (needs previous point, e.g. size 0 means disabled).
  • update documentation.

comment:9 Changed 18 months ago by tomek

  • Resolution set to complete
  • Status changed from accepted to closed

comment:10 Changed 17 months ago by fdupont

  • Resolution complete deleted
  • Status changed from closed to reopened

comment:11 Changed 17 months ago by fdupont

  • Resolution set to duplicate
  • Status changed from reopened to closed
Note: See TracTickets for help on using tickets.