[patch] Adding fq_codel to Gargoyle

Discuss the technical details of Gargoyle and ongoing development

Moderator: Moderators

Post Reply
Posts: 5
Joined: Sun May 26, 2013 3:42 pm

[patch] Adding fq_codel to Gargoyle

Post by lioux »


I think I should share this patch. It's a naive addition of fq_codel to Gargoyle. I simply modified /etc/init.d/qos_gargoyle to use fq_codel instead of sfq.

I call it naive because I did little tuning. I tried to maintain the previous tuning for sfq so that the modification could be easily identified on the patch.

1) I arbitrarily chose a byte queue limit of 3000. It should mean "2 packets". This is in line with David Taht's scripts.
2) My line has an average RTT of 40ms so that's why I use it as a target but you can remove that and take your chances.
3) I have a PPPoE LLC/SNAP connection so my overhead is 38. I don't have the extra 2 for padding because my internal LAN mtu is 1446 which should make it a multiple of 53 (fixed ATM cell size).

I got the overhead figure and the MTU one from:
a) http://sd.wareonearth.com/~phil/net/overhead/
b) http://ace-host.stuart.id.au/russell/files/tc/tc-atm/

My estimates may be wrong. :) Let me know if I made a mistake.

This allows you to use fq_codel. However, it is not correctly tuned so it could underperform for you. For me, it's giving similar results to sfq and a little better overall latency. The cost? It's using A LOT of cpu. My TP-Link TL-4300 is averaging 40% of CPU load. sfq averaged less than 5%. I don't mind that the router is using 40% because that's what it's there for. However, I find the difference alarming.

I hope it's useful and I look forward to your input on tuning fq_codel.

Vanilla Gargoyle already does an excellent job at fighting bufferbloat. fq_codel would be just adding up.
Unified patch to /etc/init.d/qos_gargoyle to use fq_codel
(1.46 KiB) Downloaded 330 times

Posts: 8
Joined: Sat Mar 30, 2013 5:55 pm

Re: [patch] Adding fq_codel to Gargoyle

Post by dtaht »

A couple comments.

HTB's ADSL and encapsulation calculation is indeed broken, possibly
for quite a long time. There are people working on it but so far no
resolution. So your settings for overhead are probably not accomplishing a lot.

The codel "target" has nothing to do with your RTT. It's the amount of packets stacked up you are willing to tolerate as local delay.

A target above what it takes to get a single packet out of your rate limiter seems to be close to the right thing. Kathie and Van maintain that even at low rates it doesn't need to be changed from the 5ms default, reality seems to be a bit different but we continue to work on it. (at the moment most of our issues on ADSL seem to have come from htb, not codel)

The byte queue limit is for devices that are running at wire rate, not using a soft rate limiter like htb or hfsc. A byte queue limit of 1514 is about right at a 10Mbit line rate, 3000 is about right at 100Mbit on most of the hardware I've benchmarked (but it's worth benchmarking), and at gigE leave it at the defaults. And there's no need to fiddle with it if you are using soft rate limiting.

I find the 40% difference in CPU usage on your tp-4300 alarming too, there should be very little difference in CPU usage between sfq and fq_codel. For example, at least on the ar71xx chipsets I mostly use there is NO performance difference between pfifo_fast and fq_codel at 300+Mbit.

An oprofile would be interesting. The only thing I can see causing a huge hit like that is a very suboptimal timestamping facility, but it would be best to profile.

The numbers I have show fq_codel better than sfq at higher bandwidths (2Mbit + ) and at higher numbers of streams, such as those imposed by the RRUL tests.

SFQ is quite good below that however. I would certainly like to get around to benchmarking gargoyles system soon.

dave taht

Posts: 8
Joined: Sat Mar 30, 2013 5:55 pm

Re: [patch] Adding fq_codel to Gargoyle

Post by dtaht »

I looked over your patches. I'm not sure what package where you are patching (?)

oh, cool, the existing qos system uses sfqred (fq_codel's predecessor). I was quite proud of that til codel.

As for this line:

tc qdisc add dev $qos_interface parent 1:$next_class_index handle $next_class_index:1 fq_codel limit $(($tbw/500)) flows 256 quantum 500 target $((${AVERAGE_RTT}*1000)) ecn

I don't know what tbw is but you want several hundred packets at least, I've settled on 600 or so at bandwidths below 20Mbit. There is no real need to limit flows to 256, the default of 1024 is fine. Openwrt went with a quantum of 300 based on the cablelabs study. As for the target, that can't possibly be right. Try the default on that please.

Posts: 250
Joined: Thu Jan 17, 2013 11:43 pm

Re: [patch] Adding fq_codel to Gargoyle

Post by BashfulBladder »

lioux wrote:My TP-Link TL-4300 is averaging 40% of CPU load.

Just WOW. And its no slouch of a router.

And thanks for your investment in time to making it work.
TP-Link WDR3600 v1.1 running 1.5.10+ L10n-English (Built 20130922 - OpenWrt r38093)
TP-Link WDR4300 running 1.5.10+ i18n-English (Built 20131010 - OpenWrt r38286)


Posts: 1371
Joined: Fri Aug 21, 2009 5:09 pm

Re: [patch] Adding fq_codel to Gargoyle

Post by pbix »

dtaht wrote:oh, cool, the existing qos system uses sfqred (fq_codel's predecessor). I was quite proud of that til codel.

Actually Gargoyle uses HFSC and SFQ. There was a short period in which I used sfqred but I have long since replaced it with sfq and flow to provide better performance and automatic bandwidth sharing by IP.

dtaht wrote:I would certainly like to get around to benchmarking gargoyles system soon.

I look forward to any results you might get in your benchmarking. Especially at <1Mbps on DSL links using the appropriate stab settings. I am intrigued by the concept of fq_codel and interested to see if any meaningful improvement could be made in Gargoyle QoS with it. Please post again.
Linksys WRT1900ACv2
Netgear WNDR3700v2
TP Link 1043ND v3
TP-Link TL-WDR3600 v1
Buffalo WZR-HP-G300NH2

Posts: 8
Joined: Sat Mar 30, 2013 5:55 pm

Re: [patch] Adding fq_codel to Gargoyle

Post by dtaht »

Well, you are actually using the sfqred codebase as the flows and depth parameters were added during the sfqred development process between linux 3.3 and linux 3.5. Also how perturbation was done was enhanced.

It's not anywhere near as limited as SFQ was prior to linux 3.3.

But: We found it very difficult to tune the red component regardless, and found without RED that as we added flows and increased depth to handle high rates (say, 50Mbit+) , that we incurred tail drop issues with services like youtube, and didn't work worth beans on wifi.

Thankfully shortly afterwards fq_codel showed up and we haven't touched sfq since. Still working on wifi tho.

I have looked over ACC and like it, and one of these days I'll get around to making ti work with one or more of cero's default shapers. A flaw is that at least on cable modems you have overbuffering in BOTH directions, not just egress, and on verizon, the overbuffering is on ingress (google for 'rrul rogues gallery for examples of fios and comcast performance'). On DSL the problem is largely on egress (so far as I know), so I imagine acc works very well in that environment.

(I note that there was some major breakage in htb between kernel 3.8 and now for dsl that is still being straightened out)

So I would very much like to do a head to head comparison of the two in a variety of scenarios, and have been setting up a lab to do so where I'll commence tests starting late august.

It would make me happy if I could just install the default gargoyle firmware and be able to swtich between sfq and fq_codel for that testing. (hint, hint)

Note: I generally don't have time for web forums, I'm sorry I missed last month's reply, I ticked the checkbox for email notifications this time.

Post Reply