Racoon and setkey problems

[Misak Khachatryan]

Hello there,

I 4 machines with ipsec confingured by racoon and running well by
several years. A three week ago 3 of them starting to fill the log
with messages like this:

Feb 19 10:17:57 rtr-1 racoon: [10.1.0.2] ERROR: failed to process ph2
packet (side: 1, status: 8).
          Feb 19 10:17:57 rtr-1 racoon: [10.1.0.2] ERROR: phase2
negotiation failed.
Feb 19 10:17:58 rtr-1 racoon: ERROR: libipsec failed send update (No
buffer space available)
Feb 19 10:17:58 rtr-1 racoon: ERROR: pfkey update failed.
Feb 19 10:17:58 rtr-1 racoon: [10.0.0.2] ERROR: failed to process ph2
packet (side: 0, status: 8).
Feb 19 10:17:58 rtr-1 racoon: [10.0.0.2] ERROR: phase2 negotiation failed.
Feb 19 10:18:00 rtr-1 racoon: ERROR: libipsec failed send update (No
buffer space available)
Feb 19 10:18:00 rtr-1 racoon: ERROR: pfkey update failed.

I see also increasing counter of "messages with memory allocation
failure" on "sent to userland" part.

# netstat -s -p pfkey
pfkey:
       3067523 requests sent from userland
       453974456 bytes sent from userland
       histogram by message type:
               getspi: 1533688
               update: 1533640
               add: 25
               delete: 1
               acquire: 42
               register: 16
               flush: 10
               dump: 18
               x_promisc: 23
               x_spdadd: 48
               x_spddump: 5
               x_spdflush: 7
       0 messages with invalid length field
       0 messages with invalid version field
       0 messages with invalid message type field
       0 messages too short
       0 messages with memory allocation failure
       0 messages with duplicate extension
       0 messages with invalid extension type
       0 messages with invalid sa type
       0 messages with invalid address extension
       7717719 requests sent to userland
       1461098984 bytes sent to userland
       histogram by message type:
               getspi: 1533688
               update: 1533640
               add: 25
               delete: 1
               acquire: 1569975
               register: 16
               expire: 2968244
               flush: 10
               dump: 111982
               x_promisc: 48
               x_spdadd: 48
               x_spddump: 60
               x_spdflush: 7
       1757766 messages toward single socket
       1533864 messages toward all sockets
       9076534 messages toward registered sockets
       1644111 messages with memory allocation failure

3 of machines running   10.4-RELEASE-p1, one 10.3.
Two of the machine almost the same, only ip addresses and few lines of
configs differ. One is OK, other one have problem.

Running almost any setkey command leads to:

 # setkey -x
setkey: send: No buffer space available

All packet versions are completely the same, binaries exactly same size.

Any help will be appreciated.

Best regards,
Misak Khachatryan