Found a way to break QOS?

Report issues relating to bandwith monitoring, bandwidth quotas or QoS in this forum.

Moderator: Moderators

dsalch
Posts: 123
Joined: Wed Jan 06, 2016 2:09 pm
Location: Tx

Re: Found a way to break QOS?

Post by dsalch »

root@Gargoyle:~# /etc/init.d/qos_gargoyle restart
+ + + tcgrep hfsc qdisc show
awk '{print $5}'

+ tc qdisc del dev eth0.2 root
+ tc qdisc del dev imq0 root
+ delete_chain_from_table mangle qos_egress
+ delete_chain_from_table mangle qos_ingress
+ set +x
+ wait_sec=15
+ uci -P /var/state get network.wan.up
+ '[' -z 1 ]
+ uci -P /var/state get network.wan.ifname
+ qos_interface=eth0.2
+ '[' -z eth0.2 ]
+ ifconfig eth0.2
+ sed -n 's/.*inet addr:\([0-9/.]*\).*/\1/p'
+ wan_ip=192.168.254.65
+ + sed -n 's/.*inet addr:\([0-9/.]*\).*/\1/p'
ifconfig br-lan
+ local_ip=192.168.3.1
+ uci -P /var/state get network.vpn.ifname
+ vpn_interface=
+ set +x
+ '[' -n 1600 ]
+ total_upload_bandwidth=1600
+ upload_default_class=uclass_3
+ total_bandwidth=
+ default_class=
+ set +x
+ '[' -n 24000 ]
+ total_download_bandwidth=24000
+ download_default_class=dclass_3
+ overhead='stab linklayer atm overhead 32 mtu 2048 '
+ uci get network.wan.proto
+ wan_proto=static
+ '[' static '!=' pppoe ]
+ '[' 1600 -ge 1100 ]
+ '[' 24000 -ge 3100 ]
+ overhead=
+ set +x
+ tc qdisc add dev eth0.2 root handle 1:0 hfsc default 1
+ tc class add dev eth0.2 parent 1:0 classid 1:1 hfsc ls rate 1000Mbit ul rate 1600kbit
+ set +x
+ eval echo '$uclass_1'
+ echo 12 -1 -1
+ uclass_def='12 -1 -1'
+ + echo 12 -1 -1
awk ' {print $1}'
+ m2=120
+ echo 12 -1 -1+
awk ' {print $3}'
+ min_bandwidth=-1
+ '[' -1 -gt 0 ]
+ ll_str=
+ + awkecho ' {print $2}' 12
-1 -1
+ max_bandwidth=-1
+ '[' -1 -ge 0 ]
+ max_bandwidth=0
+ ul_str=
+ tbw=0
+ '[' 0 -lt 6000 ]
+ tbw=6000
+ tc class add dev eth0.2 parent 1:1 classid 1:2 hfsc ls m2 120Mbit
+ tc qdisc add dev eth0.2 parent 1:2 handle 2:1 sfq headdrop limit 24 divisor 256
+ tc filter add dev eth0.2 parent 1:0 protocol ip handle 0x2 fw flowid 1:2
+ tc filter add dev eth0.2 parent 2: handle 1 flow divisor 256 map key nfct-src and 0xff
+ set +x
+ eval echo '$uclass_2'
+ echo 12 -1 -1
+ uclass_def='12 -1 -1'
+ + echo 12 -1 -1awk ' {print $1}'

+ m2=120
+ + awk ' {print $3}'
echo 12 -1 -1
+ min_bandwidth=-1
+ '[' -1 -gt 0 ]
+ ll_str=
+ + awk ' {print $2}'
echo 12 -1 -1
+ max_bandwidth=-1
+ '[' -1 -ge 0 ]
+ max_bandwidth=0
+ ul_str=
+ tbw=0
+ '[' 0 -lt 6000 ]
+ tbw=6000
+ tc class add dev eth0.2 parent 1:1 classid 1:3 hfsc ls m2 120Mbit
+ tc qdisc add dev eth0.2 parent 1:3 handle 3:1 sfq headdrop limit 24 divisor 256
+ tc filter add dev eth0.2 parent 1:0 protocol ip handle 0x3 fw flowid 1:3
+ tc filter add dev eth0.2 parent 3: handle 1 flow divisor 256 map key nfct-src and 0xff
+ set +x
+ eval echo '$uclass_3'
+ echo 12 -1 -1
+ uclass_def='12 -1 -1'
+ + awk ' {print $1}'
echo 12 -1 -1
+ m2=120
+ + awk ' {print $3}'echo
12 -1 -1
+ min_bandwidth=-1
+ '[' -1 -gt 0 ]
+ ll_str=
+ + awk ' {print $2}'
echo 12 -1 -1
+ max_bandwidth=-1
+ '[' -1 -ge 0 ]
+ max_bandwidth=0
+ ul_str=
+ tbw=0
+ '[' 0 -lt 6000 ]
+ tbw=6000
+ tc class add dev eth0.2 parent 1:1 classid 1:4 hfsc ls m2 120Mbit
+ tc qdisc add dev eth0.2 parent 1:4 handle 4:1 sfq headdrop limit 24 divisor 256
+ tc filter add dev eth0.2 parent 1:0 protocol ip handle 0x4 fw flowid 1:4
+ tc filter add dev eth0.2 parent 4: handle 1 flow divisor 256 map key nfct-src and 0xff
+ set +x
+ eval echo '$uclass_4'
+ echo 64 -1 -1
+ uclass_def='64 -1 -1'
+ + awk ' {print $1}'
echo 64 -1 -1
+ m2=640
+ + awk ' {print $3}'
echo 64 -1 -1
+ min_bandwidth=-1
+ '[' -1 -gt 0 ]
+ ll_str=
+ + awk ' {print $2}'
echo 64 -1 -1
+ max_bandwidth=-1
+ '[' -1 -ge 0 ]
+ max_bandwidth=1
+ ul_str=
+ tbw=12
+ '[' 12 -lt 6000 ]
+ tbw=6000
+ tc class add dev eth0.2 parent 1:1 classid 1:5 hfsc ls m2 640Mbit
+ tc qdisc add dev eth0.2 parent 1:5 handle 5:1 sfq headdrop limit 24 divisor 256
+ tc filter add dev eth0.2 parent 1:0 protocol ip handle 0x5 fw flowid 1:5
+ tc filter add dev eth0.2 parent 5: handle 1 flow divisor 256 map key nfct-src and 0xff
+ set +x
+ tc qdisc change dev eth0.2 root handle 1:0 hfsc default 4
+ iptables -t mangle -N qos_egress
+ iptables -t mangle -A POSTROUTING -o eth0.2 -j qos_egress
+ set +x
+ iptables -t mangle -I qos_egress -p udp -s 192.168.3.25 --dport 3478:3479 -j MARK --set-mark 0x5
+ set +x
+ iptables -t mangle -I qos_egress -p tcp -s 192.168.3.25 --dport 8801:8802 -j MARK --set-mark 0x5
+ set +x
+ iptables -t mangle -I qos_egress -p udp -s 192.168.3.25 --dport 8801:8810 -j MARK --set-mark 0x5
+ set +x
+ iptables -t mangle -I qos_egress -s 192.168.3.99 -j MARK --set-mark 0x3
+ set +x
+ iptables -t mangle -I qos_egress -s 192.168.3.98 -j MARK --set-mark 0x2
+ set +x
+ iptables -t mangle -I qos_egress -s 192.168.3.5 -j MARK --set-mark 0x3
+ set +x
+ iptables -t mangle -I qos_egress -s 192.168.3.6 -j MARK --set-mark 0x2
+ set +x
+ iptables -t mangle -I qos_egress -s 192.168.3.7 -j MARK --set-mark 0x2
+ set +x
+ iptables -t mangle -I qos_egress -j MARK --set-mark 0x4
+ iptables -t mangle -I qos_egress -m mark '!' --mark 0x0 -j RETURN
+ iptables -t mangle -I qos_egress -m mark '!' --mark 0x0 -j CONNMARK --save-mark --mask 0x007F
+ iptables -t mangle -A qos_egress -j CONNMARK --save-mark --mask 0x007F
+ '[' 24000 -eq 0 ]
+ '[' 1600 -eq 0 ]
+ '[' 24000 -ge 0 ]
+ ip link set imq0 up
+ tc qdisc add dev imq0 root handle 1:0 hfsc default 1
+ tc class add dev imq0 parent 1:0 classid 1:1 hfsc ls rate 1000Mbit ul m2 24000kbit
+ set +x
+ class_mark_list='dclass_1:0x200 '
+ eval echo '$dclass_1'
+ echo 33 -1 -1
+ dclass_def='33 -1 -1'
+ + echoawk 33 ' {print $1}' -1
-1
+ m2=330
+ + awk ' {print $2}'
echo 33 -1 -1
+ max_bandwidth=-1
+ ul_str=
+ '[' -1 -ge 0 ]
+ max_bandwidth=24000
+ echo 33 -1 -1
+ awk ' {print $4}'
+ minRTT=
+ '[' '=' Yes ]
+ ll_str=' ls m2 330Mbit'
+ awk ' {print $3}'
+ echo 33 -1 -1
+ min_bandwidth=-1
+ rt_str=
+ '[' -1 -gt 0 ]
+ tbw=300000
+ '[' 300000 -lt 10000 ]
+ tc class add dev imq0 parent 1:1 classid 1:2 hfsc ls m2 330Mbit
+ tc qdisc add dev imq0 parent 1:2 handle 2:1 sfq headdrop limit 1200 divisor 256
+ tc filter add dev imq0 parent 1:0 prio 2 protocol ip handle 0x200 fw flowid 1:2
+ tc filter add dev imq0 parent 2: handle 1 flow divisor 256 map key dst and 0xff
+ set +x
+ class_mark_list='dclass_1:0x200 dclass_2:0x300 '
+ eval echo '$dclass_2'
+ echo 33 -1 -1
+ dclass_def='33 -1 -1'
+ + awk ' {print $1}'echo 33 -1
-1
+ m2=330
+ echo 33 -1 -1
+ awk ' {print $2}'
+ max_bandwidth=-1
+ ul_str=
+ '[' -1 -ge 0 ]
+ max_bandwidth=24000
+ + echo 33 -1 -1
awk ' {print $4}'
+ minRTT=
+ '[' '=' Yes ]
+ ll_str=' ls m2 330Mbit'
+ + awk ' {print $3}'
echo 33 -1 -1
+ min_bandwidth=-1
+ rt_str=
+ '[' -1 -gt 0 ]
+ tbw=300000
+ '[' 300000 -lt 10000 ]
+ tc class add dev imq0 parent 1:1 classid 1:3 hfsc ls m2 330Mbit
+ tc qdisc add dev imq0 parent 1:3 handle 3:1 sfq headdrop limit 1200 divisor 256
+ tc filter add dev imq0 parent 1:0 prio 3 protocol ip handle 0x300 fw flowid 1:3
+ tc filter add dev imq0 parent 3: handle 1 flow divisor 256 map key dst and 0xff
+ set +x
+ class_mark_list='dclass_1:0x200 dclass_2:0x300 dclass_3:0x400 '
+ eval echo '$dclass_3'
+ echo 22 -1 -1
+ dclass_def='22 -1 -1'
+ + awk ' {print $1}'
echo 22 -1 -1
+ m2=220
+ echo 22+ -1 -1
awk ' {print $2}'
+ max_bandwidth=-1
+ ul_str=
+ '[' -1 -ge 0 ]
+ max_bandwidth=24000
+ + awkecho 22 -1 -1 ' {print $4}'

+ minRTT=
+ '[' '=' Yes ]
+ ll_str=' ls m2 220Mbit'
+ + echo 22awk ' {print $3}' -1
-1
+ min_bandwidth=-1
+ rt_str=
+ '[' -1 -gt 0 ]
+ tbw=300000
+ '[' 300000 -lt 10000 ]
+ tc class add dev imq0 parent 1:1 classid 1:4 hfsc ls m2 220Mbit
+ tc qdisc add dev imq0 parent 1:4 handle 4:1 sfq headdrop limit 1200 divisor 256
+ tc filter add dev imq0 parent 1:0 prio 4 protocol ip handle 0x400 fw flowid 1:4
+ tc filter add dev imq0 parent 4: handle 1 flow divisor 256 map key dst and 0xff
+ set +x
+ class_mark_list='dclass_1:0x200 dclass_2:0x300 dclass_3:0x400 dclass_4:0x500 '
+ eval echo '$dclass_4'
+ echo 11 -1 3000 Yes
+ dclass_def='11 -1 3000 Yes'
+ + awk ' {print $1}'
echo 11 -1 3000 Yes
+ m2=110
+ echo 11 -1 3000 Yes
+ awk ' {print $2}'
+ max_bandwidth=-1
+ ul_str=
+ '[' -1 -ge 0 ]
+ max_bandwidth=24000
+ + awk ' {print $4}'
echo 11 -1 3000 Yes
+ minRTT=Yes
+ '[' Yes '=' Yes ]
+ d1=1
+ ll_str=' ls m1 220Mbit d 1ms m2 110Mbit'
+ + awkecho 11 -1 3000 Yes
' {print $3}'
+ min_bandwidth=3000
+ rt_str=
+ '[' 3000 -gt 0 ]
+ '[' Yes '=' Yes ]
+ d2=1
+ rt_str=' rt m1 24000kbit d 1ms m2 3000kbit'
+ tbw=300000
+ '[' 300000 -lt 10000 ]
+ tc class add dev imq0 parent 1:1 classid 1:5 hfsc rt m1 24000kbit d 1ms m2 3000kbit ls m1 220Mbit d 1ms m2 110Mbit
+ tc qdisc add dev imq0 parent 1:5 handle 5:1 sfq headdrop limit 1200 divisor 256
+ tc filter add dev imq0 parent 1:0 prio 5 protocol ip handle 0x500 fw flowid 1:5
+ tc filter add dev imq0 parent 5: handle 1 flow divisor 256 map key dst and 0xff
+ set +x
+ tc qdisc change dev imq0 root handle 1:0 hfsc default 4
+ iptables -t mangle -N qos_ingress
+ iptables -t mangle -A FORWARD -i eth0.2 -j qos_ingress
+ iptables -t mangle -A INPUT -i eth0.2 -j qos_ingress
+ set +x
+ iptables -t mangle -I qos_ingress -p tcp --sport 8801:8802 -d 192.168.3.25 -j MARK --set-mark 0x500
+ set +x
+ iptables -t mangle -I qos_ingress -p udp --sport 3478:3479 -d 192.168.3.25 -j MARK --set-mark 0x500
+ set +x
+ iptables -t mangle -I qos_ingress -p udp --sport 8801:8810 -d 192.168.3.25 -j MARK --set-mark 0x500
+ set +x
+ iptables -t mangle -I qos_ingress -d 192.168.3.98 -j MARK --set-mark 0x200
+ set +x
+ iptables -t mangle -I qos_ingress -d 192.168.3.99 -j MARK --set-mark 0x300
+ set +x
+ iptables -t mangle -I qos_ingress -d 192.168.3.7 -j MARK --set-mark 0x200
+ set +x
+ iptables -t mangle -I qos_ingress -d 192.168.3.6 -j MARK --set-mark 0x200
+ set +x
+ iptables -t mangle -I qos_ingress -d 192.168.3.5 -j MARK --set-mark 0x300
+ set +x
+ iptables -t mangle -I qos_ingress -j MARK --set-mark 0x400
+ iptables -t mangle -I qos_ingress -m mark '!' --mark 0x0 -j RETURN
+ iptables -t mangle -I qos_ingress -m mark '!' --mark 0x0 -j CONNMARK --save-mark --mask 0x7F00
+ iptables -t mangle -I qos_ingress -j IMQ --todev 0
+ iptables -t mangle -A qos_ingress -j CONNMARK --save-mark --mask 0x7F00
+ set +x
+ '[' -z 8.8.8.8 ]
+ iptables -t mangle -I qos_ingress -p icmp --icmp-type 0 -s 8.8.8.8 -j RETURN
+ tc class add dev eth0.2 parent 1:1 classid 1:127 hfsc rt umax 106 dmax 10ms rate 4kbit
+ tc qdisc add dev eth0.2 parent 1:127 pfifo
+ tc filter add dev eth0.2 parent 1:0 prio 1 protocol ip handle 127 fw flowid 1:127
+ iptables -t mangle -I qos_egress -p icmp --icmp-type 8 -s 192.168.254.65 -d 8.8.8.8 -j MARK --set-mark 127
+ '[' -n ]
+ pinglimit=11
+ qosmon -a -b 800 8.8.8.8 24000 11
+ set +x
root@Gargoyle:~#

dsalch
Posts: 123
Joined: Wed Jan 06, 2016 2:09 pm
Location: Tx

Re: Found a way to break QOS?

Post by dsalch »

Ok, new info.

Now I wonder if this whole thing is in fact a cpu overload.

I set up a very consistent test with a single stream in each class, while watching TOP on the console. In summary, it seems that QOS breaks down whenever SIRQ reaches above 55% for any reason. at 60% SIRQ it seems that QOS drops out completely and cant recover until traffic pretty much ends totally. At 50% to 55% SIRQ QOS seems to drift in and out of working.

MinBW does seem to dramatically increase CPU load over having just class percentages.

Multiple streams also increase CPU over single streams per class, even at the same total bandwidth.

With max BW set at 22,000kbps, QOS works fine with a single stream per class. with two streams in a class it starts to break.

With MaxBW at 23,000 QOS drifts in and out of working

With maxBW at 24000 QOS quits completely.

These BW numbers seem to track SIRQ percentages as described above.

pbix
Developer
Posts: 1373
Joined: Fri Aug 21, 2009 5:09 pm

Re: Found a way to break QOS?

Post by pbix »

You say that you have a DSL link but according to this Wiki
https://en.wikipedia.org/wiki/Asymmetri ... riber_line DSL cannot achieve 30Mbit download. So that is a confusion point and setting your QoS download speed above 24Mbit may itself break QoS. Your QoS router needs to be the slowest link. What testing have you done to verify that this link can indeed deliver 30Mbps?

Anyway from the listing you posted Gargoyle has not switched you to DSL mode because the speed you entered are higher than is expected for DSL. If you are sure you have DSL edit the file /etc/init.d/qos_gargoyle as follows to make it switch.

Line 353

Code: Select all

if [ "$wan_proto" != "pppoe" ] && [ "$total_upload_bandwidth" -ge 1100 ] && [ "$total_download_bandwidth" -ge 3100 ] ; then
Change to

Code: Select all

if [ "$wan_proto" != "pppoe" ] && [ "$total_upload_bandwidth" -ge 3300 ] && [ "$total_download_bandwidth" -ge 3100 ] ; then
This change will make for a more stable and reliable QoS. I have seen dramatic improvements myself in my networks.

Regarding CPU loads, your 1043ND router should be able to handle 30Mbps with the default QoS. Custom settings like yours do effect performance but not that much in my experience.
Linksys WRT1900ACv2
Netgear WNDR3700v2
TP Link 1043ND v3
TP-Link TL-WDR3600 v1
Buffalo WZR-HP-G300NH2
WRT54G-TM

dsalch
Posts: 123
Joined: Wed Jan 06, 2016 2:09 pm
Location: Tx

Re: Found a way to break QOS?

Post by dsalch »

Sorry, more missing info...

The 30mbps is 2 bonded 16mbps lines. The bonding is done at the ISP, and works well. At the DSL router I can achieve a very consistent 31 to 32mbps on a single download thread.

And of course, the 2.2mbps download is the bonding of two 1.1mbps lines.

https://drive.google.com/open?id=1NtAGn ... l2c1DcVAS6

dsalch
Posts: 123
Joined: Wed Jan 06, 2016 2:09 pm
Location: Tx

Re: Found a way to break QOS?

Post by dsalch »

You're a genius!

Making that change to "switch to dsl mode" worked like a charm! A quick tests now shows that QOS is rock solid up to about 27mbps download, even at 60% to 65% sirq. I tried with and without the special video class, and it performs as expected.

27mbps is the effective consistent limit that I can get without QOS, so this is as good as it gets :)

Curious, I didnt know there was a "dsl mode" in this software? Any particular reason that it isn't exposed as a switch?

Daeron
Posts: 28
Joined: Mon Oct 31, 2016 5:30 am

Re: Found a way to break QOS?

Post by Daeron »

Pbix, am I right to assume that said code changed across Gargoyle versions? I'm currently running 1.10.0 on a TP-Link TL-WR1043N/ND v1, but can't find the relevant code in the mentioned file. I have a 30/3 line so I'd like to try it myself to see if there are any improvements.

This also makes me wonder what version dsalch is running, as multiple people reported that 1.11.x is pretty hard on the older 1043 revisions (although the V2 seems more capable). Could explain the CPU loads. I also saw ACC occasionally being pretty hard on my (most likely) already taxed CPU. With that said 30/3(-ish) is definitely doable with QoS enabled.

To answer some of the questions, my understanding is that setting minimum bandwidth is supposed to be identical to bandwidth% (assuming matching values), with the only meaningful difference being that ACC will not scale the former down when the total bandwidth is adjusted. But there are more qualified people than me to answer that.

dsalch
Posts: 123
Joined: Wed Jan 06, 2016 2:09 pm
Location: Tx

Re: Found a way to break QOS?

Post by dsalch »

I've been watching this run all day and it worked flawlessly under all conditions. Ended up getting qos Max at 27000/1600.

Interestingly, the CPU utilization also decreased noticably under full load. Maybe 10 to 20% .

I'm happy enough at this point to try introducing min bw again tomorrow and see if it holds.

I am using 1.11.x, upgraded about a couple months ago.

dsalch
Posts: 123
Joined: Wed Jan 06, 2016 2:09 pm
Location: Tx

Re: Found a way to break QOS?

Post by dsalch »

UPDATE:

I tested further today. After a full day of working under all conditions with nothing but shared % of bandwidth, I re-activatred the min bw.

By setting a min BW on the salchstr class, at a level equal to that of the max % at full bw, it broke. a full stream in any other class then kills tcp sessions in the remaining classes.

IF i activate the minBW on download salchstr class, nothing changes. It operates identically to the max% bandwidth.

So, for some reason, setting a minBW is somehow different than setting the equal % max bw for a class in the upload direction.

Thoughts?

UPDATE: Interesting observation / clarification.

It is the presence of the minBW setting that seems to cause this, not traffic. In other words, when the min BW is set on an upload class, that setting seems to allow a second class to swamp bandwidth for a third class, even if the actual throughput in the first minbw class is minimal.

dsalch
Posts: 123
Joined: Wed Jan 06, 2016 2:09 pm
Location: Tx

Re: Found a way to break QOS?

Post by dsalch »

Another update. I started testing the upload QOS action itself and found an anomaly.

The kbps settings on the upload QOS screen seems to be off somehow.

In order to get 1.6mbps to pass upload, I must set the maxBW setting on the QOS screen to 2300kbps. The chart on the bandwidth usage screen matches what the actual throughput is, but the class BW readings and settings on the QOS screen do not seem to match.

With the setting of max BW on upload QOS that allows actual max throughput (about 700kbps difference) the numbers in the real time QOS class box show a throughput that is impossible on my actual line. So it appears that QOS upload somehow is registering and setting about 700kbps higher than it really it.

How is this possible?

In light of this, the settings I was using on upload QOS were starving the tcp streams, not enough to allow ACK at the rate of download. Not sure how that would affect the minBW problem I experience, but seems significant.

pbix
Developer
Posts: 1373
Joined: Fri Aug 21, 2009 5:09 pm

Re: Found a way to break QOS?

Post by pbix »

dsalch wrote:You're a genius!
I didnt know there was a "dsl mode" in this software? Any particular reason that it isn't exposed as a switch?
Yes, yes I am :-) A switch might be a good addition to the interface. It used to be that it was easy to automatically detect so it was not there in the beginning. Recently things have gotten more complex.
Daeron wrote:Pbix, am I right to assume that said code changed across Gargoyle versions? I'm currently running 1.10.0 on a TP-Link TL-WR1043N/ND v1, but can't find the relevant code in the mentioned file.
This line of code is new for v1.11. v1.11 does a better job of detecting DSL connections than previous versions, although it is still not perfect as this thread demonstrates. You should upgrade and change this line as well and report your results.

The idea that performance is impacted by this change seems unlikely to me even though statements have been made here to that effect. You cannot compare CPU load of a system that does not work to one that does. In my experience the only thing which will make a difference in performance will be a change in the underlying OpenWrt version or complexity in your QoS rules.
Linksys WRT1900ACv2
Netgear WNDR3700v2
TP Link 1043ND v3
TP-Link TL-WDR3600 v1
Buffalo WZR-HP-G300NH2
WRT54G-TM

Post Reply