[patch] Adding fq_codel to Gargoyle

[lioux]

Hi,

I think I should share this patch. It's a naive addition of fq_codel to Gargoyle. I simply modified /etc/init.d/qos_gargoyle to use fq_codel instead of sfq.

I call it naive because I did little tuning. I tried to maintain the previous tuning for sfq so that the modification could be easily identified on the patch.

1) I arbitrarily chose a byte queue limit of 3000. It should mean "2 packets". This is in line with David Taht's scripts.
2) My line has an average RTT of 40ms so that's why I use it as a target but you can remove that and take your chances.
3) I have a PPPoE LLC/SNAP connection so my overhead is 38. I don't have the extra 2 for padding because my internal LAN mtu is 1446 which should make it a multiple of 53 (fixed ATM cell size).

I got the overhead figure and the MTU one from:
a) http://sd.wareonearth.com/~phil/net/overhead/
b) http://ace-host.stuart.id.au/russell/files/tc/tc-atm/

My estimates may be wrong. Let me know if I made a mistake.

This allows you to use fq_codel. However, it is not correctly tuned so it could underperform for you. For me, it's giving similar results to sfq and a little better overall latency. The cost? It's using A LOT of cpu. My TP-Link TL-4300 is averaging 40% of CPU load. sfq averaged less than 5%. I don't mind that the router is using 40% because that's what it's there for. However, I find the difference alarming.

I hope it's useful and I look forward to your input on tuning fq_codel.

Vanilla Gargoyle already does an excellent job at fighting bufferbloat. fq_codel would be just adding up.

[dtaht]

A couple comments.

HTB's ADSL and encapsulation calculation is indeed broken, possibly
for quite a long time. There are people working on it but so far no
resolution. So your settings for overhead are probably not accomplishing a lot.

The codel "target" has nothing to do with your RTT. It's the amount of packets stacked up you are willing to tolerate as local delay.

A target above what it takes to get a single packet out of your rate limiter seems to be close to the right thing. Kathie and Van maintain that even at low rates it doesn't need to be changed from the 5ms default, reality seems to be a bit different but we continue to work on it. (at the moment most of our issues on ADSL seem to have come from htb, not codel)

The byte queue limit is for devices that are running at wire rate, not using a soft rate limiter like htb or hfsc. A byte queue limit of 1514 is about right at a 10Mbit line rate, 3000 is about right at 100Mbit on most of the hardware I've benchmarked (but it's worth benchmarking), and at gigE leave it at the defaults. And there's no need to fiddle with it if you are using soft rate limiting.

I find the 40% difference in CPU usage on your tp-4300 alarming too, there should be very little difference in CPU usage between sfq and fq_codel. For example, at least on the ar71xx chipsets I mostly use there is NO performance difference between pfifo_fast and fq_codel at 300+Mbit.

An oprofile would be interesting. The only thing I can see causing a huge hit like that is a very suboptimal timestamping facility, but it would be best to profile.

The numbers I have show fq_codel better than sfq at higher bandwidths (2Mbit + ) and at higher numbers of streams, such as those imposed by the RRUL tests.

SFQ is quite good below that however. I would certainly like to get around to benchmarking gargoyles system soon.

dave taht

[dtaht]

I looked over your patches. I'm not sure what package where you are patching (?)

oh, cool, the existing qos system uses sfqred (fq_codel's predecessor). I was quite proud of that til codel.

As for this line:

tc qdisc add dev $qos_interface parent 1:$next_class_index handle $next_class_index:1 fq_codel limit $(($tbw/500)) flows 256 quantum 500 target $((${AVERAGE_RTT}*1000)) ecn

I don't know what tbw is but you want several hundred packets at least, I've settled on 600 or so at bandwidths below 20Mbit. There is no real need to limit flows to 256, the default of 1024 is fine. Openwrt went with a quantum of 300 based on the cablelabs study. As for the target, that can't possibly be right. Try the default on that please.

[BashfulBladder]

My TP-Link TL-4300 is averaging 40% of CPU load.

Just WOW. And its no slouch of a router.

And thanks for your investment in time to making it work.

[pbix]

oh, cool, the existing qos system uses sfqred (fq_codel's predecessor). I was quite proud of that til codel.

Actually Gargoyle uses HFSC and SFQ. There was a short period in which I used sfqred but I have long since replaced it with sfq and flow to provide better performance and automatic bandwidth sharing by IP.

I would certainly like to get around to benchmarking gargoyles system soon.

I look forward to any results you might get in your benchmarking. Especially at <1Mbps on DSL links using the appropriate stab settings. I am intrigued by the concept of fq_codel and interested to see if any meaningful improvement could be made in Gargoyle QoS with it. Please post again.

[dtaht]

Well, you are actually using the sfqred codebase as the flows and depth parameters were added during the sfqred development process between linux 3.3 and linux 3.5. Also how perturbation was done was enhanced.

It's not anywhere near as limited as SFQ was prior to linux 3.3.

But: We found it very difficult to tune the red component regardless, and found without RED that as we added flows and increased depth to handle high rates (say, 50Mbit+) , that we incurred tail drop issues with services like youtube, and didn't work worth beans on wifi.

Thankfully shortly afterwards fq_codel showed up and we haven't touched sfq since. Still working on wifi tho.

I have looked over ACC and like it, and one of these days I'll get around to making ti work with one or more of cero's default shapers. A flaw is that at least on cable modems you have overbuffering in BOTH directions, not just egress, and on verizon, the overbuffering is on ingress (google for 'rrul rogues gallery for examples of fios and comcast performance'). On DSL the problem is largely on egress (so far as I know), so I imagine acc works very well in that environment.

(I note that there was some major breakage in htb between kernel 3.8 and now for dsl that is still being straightened out)

So I would very much like to do a head to head comparison of the two in a variety of scenarios, and have been setting up a lab to do so where I'll commence tests starting late august.

It would make me happy if I could just install the default gargoyle firmware and be able to swtich between sfq and fq_codel for that testing. (hint, hint)

Note: I generally don't have time for web forums, I'm sorry I missed last month's reply, I ticked the checkbox for email notifications this time.