TCP: time wait bucket table overflow

[mix]

I have been seeing "TCP: time wait bucket table overflow" messages sporadically in the results of dmesg for the last couple 1.4 releases of Gargoyle on my WRT54GL. My guess is that some kernel setting is set a bit low. I am using the default settings for Connection Limits, and have even experimented with setting them a bit lower (4096/360/120), but still get this message. I am not running any torrent clients or anything that creates a massive amount of network connections as far as I know. Just wondering if anyone else has seen this error message?

[pbix]

How many connection show on the connection screen?

If you are getting this message I would think that increasing the maximum connection count might be the normal course of action. But if you are running out of RAM then something like this might occur and lower the limit as you have done might change the message into something else.

[mix]

You certainly know as well as I do that the number of connections changes by the second. No way I am using more than 1000 at any one time between all nat clients combined as reported by the Gargoyle web interface. This has something to do with connections in the time_wait state and the number of sockets available. There is limited info on google, though there are some references to changing the value of /proc/sys/net/ipv4/tcp_max_tw_buckets - Mine is set for 256.

There are a bunch of values in /proc/sys/net/ipv4 that relate to time_wait and time_wait buckets:

tcp_rfc1337
tcp_tw_recycle
tcp_tw_reuse

And there are many more that probably indirectly influence when these limits are reached. Luckily, /etc/sysctl.conf is pretty sparse, so it's not to hard to tell what is being set and what is defaulting from somewhere.

http://www.kernel.org/doc/Documentation ... sysctl.txt

I guess I am trying to figure out if this (256) is a reasonable number for a nat router. Maybe someone using dd-wrt or Tomato can post their values. I am not sure if this value is different depending on the amount of ram available or not. IE., my 16 mb WRT54GL is lower than someone with a new N router with 32 or 64 mb of ram.

[Eric]

mix: I had the same concerns, so in the 1.5.x branch net.ipv4.tcp_tw_reuse is set. When researching this setting several places said to be very careful with that option, as it has the potential to cause problems. That's why I only added it to the 1.5.x branch first -- though so far I haven't seen any problems.

If you want to add it to 1.4.x just edit /etc/sysctl.conf and add this line:

Code: Select all

net.ipv4.tcp_tw_reuse=1

[mix]

Eric, I added this to my config. I am still getting the error messages. Don't really know what to think...

[pbix]

I found this in ../net/ipv4/tcp_minisock.c

Code: Select all

} else {
		/* Sorry, if we're out of memory, just CLOSE this
		 * socket up.  We've got bigger problems than
		 * non-graceful socket closings.
		 */
		LIMIT_NETDEBUG(KERN_INFO "TCP: time wait bucket table overflow\n");
	}

This error occurs when the kernel fails to allocate a struct inet_timewait_sock needed to put the connection into a timewait state.

You are out of RAM memory so the only hope you have to avoid this message would be to reduce the number of connections allowed.

[hnl_dk]

...
You are out of RAM memory so the only hope you have to avoid this message would be to reduce the number of connections allowed.

@mix
just tried to check your hardware on the openwrt wiki...
if you wish to try (but it is on your own risk to try)... would you try to add a sd card to your router and setup a swap partition on it...
SD hardware hack: http://wiki.openwrt.org/toh/linksys/wrt ... mmcsd.card and http://www.allaboutjake.com/network/lin ... t54g/hack/

[mix]

What do you guys know about the modules that are set to load automatically upon boot? It sure looks to me that many of these modules in /etc/modules.d aren't always needed. They could instead be enabled or disabled based upon whether or not the appropriate corresponding setting in the web interface has been enabled, disabled, or is being actively used/populated. Wouldn't this free up some ram?

[mix]

This error occurs when the kernel fails to allocate a struct inet_timewait_sock needed to put the connection into a timewait state.

You are out of RAM memory so the only hope you have to avoid this message would be to reduce the number of connections allowed.

Out of ram, or out of allocated time wait buckets? It sure looks like this else statement only executes when tw is set to NULL. Since tw is initialized as null, this would occur when the following is false:

Code: Select all

         if (tcp_death_row.tw_count < tcp_death_row.sysctl_max_tw_buckets)
                 tw = inet_twsk_alloc(sk, state);

Now I guess inet_twsk_alloc allocates memory... and of course returns null if it can't. I can't prove which one it is without compiling the code I guess.

So I guess, back to my question from before, is 256 a realistic number? Is this what everyone with Gargoyle has as a limit? Is this what other firmware distros are using?

In the meantime I have removed some unneeded modules from memory, but left the limit the same 256. Free mem, atleast according to "free," is over 1000.

[bardu]

Maybe someone using dd-wrt or Tomato can post their values. I am not sure if this value is different depending on the amount of ram available or not. IE., my 16 mb WRT54GL is lower than someone with a new N router with 32 or 64 mb of ram.

Bucket values:

1. ASUS RT-N16 (used as small station running as php/mysql test server)

DD-WRT v24-sp2 king (c) 2012 NewMedia-NET GmbH
Release: 03/17/12 (SVN revision: 18730M)
Enhanced with OTRW

Code: Select all

root@rt-n16:~# sysctl -a | grep bucket
error: permission denied on key 'net.ipv4.route.flush'
net.ipv4.tcp_max_tw_buckets = 8192
net.ipv4.netfilter.ip_conntrack_buckets = 2048
net.netfilter.nf_conntrack_buckets = 2048

2. TP-LINK 1043ND v1.8 as main router, loaded with latest gargoyle 1.5.4

Code: Select all

root@Gargoyle:~# sysctl -a | grep bucket
net.netfilter.nf_conntrack_buckets = 1024
sysctl: error reading key 'net.ipv4.route.flush': Permission denied
net.ipv4.tcp_max_tw_buckets = 512
net.ipv4.netfilter.ip_conntrack_buckets = 1024

I see gargoyle uses a very small value for max tw buckets.
Question:
If I put net.ipv4.tcp_max_tw_buckets = 8192 in /etc/systl.conf it will change gargoyle default value for buckets or not?
I ask this cause I don't wanna reboot the router for nothing.

LE1.
Nevermind. I added the below values to /etc/sysctl.conf and rebooted the router.

Code: Select all

#the "bucket" experiment
net.ipv4.tcp_max_tw_buckets = 8192
#end bucket experiment

and voila:

Code: Select all

root@Gargoyle:~# sysctl -a | grep bucket
net.netfilter.nf_conntrack_buckets = 1024
sysctl: error reading key 'net.ipv4.route.flush': Permission denied
net.ipv4.tcp_max_tw_buckets = 8192
net.ipv4.netfilter.ip_conntrack_buckets = 1024

I got really frustrated when i could not accomplish simple openwrt svn checkout .
LE2.

Code: Select all

root@Gargoyle:~# uptime
 11:03:51 up 14:58, load average: 0.02, 0.01, 0.00

No more "bucket" messages in dmesg output!
After reading some dd-wrt wiki found here: http://www.dd-wrt.com/wiki/index.php/Router_Slowdown
I also increased max connections to 16384 in Gargoyle frontend and hashsize to 16384. To increase hashsize I used rc.local script.

Code: Select all

root@Gargoyle:~# cat /etc/rc.local
# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.
echo "16384" > /sys/module/nf_conntrack/parameters/hashsize
exit 0

All is nicer and smoother now.