Found a way to break QOS?

[dsalch]

Success!

Without setting a min BW on upload, QOS works flawlessly. The three classes no operate independently of each other without interaction.

If I turn on "min BW" for the salchstr upload class, any one of the three can swamp the bandwidth and cause tcp traffic to stall.

here is a shot of the classes working independently as desired
https://drive.google.com/open?id=1A6Zch ... 5qio1_yCLU

I dont know why it wont work with a min BW setting on upload, but everything seems to work ok without it. I just figured the percentage of total for each class as the min required, and it works ok.

[pbix]

Saturated means you are at the fair link limit computed by ACC (or manually set). So its not that ACC is active but rather that the the load is near the limit. This is independently true on the uplink and the down link.

With 30Mbit down and 1.6Mbit up your link is highly asymmetric. Is this a DSL link? If so, what version of Gargoyle are you running? Run "/etc/init.d/qos_gargoyle restart" from the command line and post the results.

To sustain your 30Mbps download will require almost all of your uplink bandwidth just for the ACK packets and very accurate accounting by QoS.

[dsalch]

Yes, this is dsl.

I wonder why it works without min be on upload but breaks with Min be on upload. The effective numbers are the same, but somehow there is a difference. Even when min bw is set lower than the % of total at saturation, it breaks tcp streams.

It sort of all makes sense now. Udp is not affected like tcp because of the tcp ack packets are what is being affected somehow.

I am happy it's working, but confused why min bw breaks it.

[Daeron]

I found multiple things about your posts rather confusing. So perhaps it would be the best to start over by clarifying things and as pbix already suggested, aim for the simplest possible config that can possibly work. Build it one step at a time and ensure each step works as intended instead of trying to do it all at once.

As far as the main router is concerned, you have 2 additional routers (ROUTER1, ROUTER2) and a PC1? Yet then you say you have another class locally which is a conglomeration of devices on your local (wifi) network? So which one is it?

How many devices are there? Which ones actually need special treatment? What are the bandwidth requirements of that special treatment? And I'd prefer using my terminology because the nicknames get confusing fast.

Keep in mind that Gargoyle QoS by default splits the available bandwidth between devices (and classes), so generally you don't need to specify minimum bandwidth for every class (the bandwith% already does that).

Also I would suggest turning off ACC during testing and simply lowering your total bandwidth to 50%. This way you have less variability outside of your control, ensures that QoS is actually low enough to work and makes it easier to saturate the link and find the problems with your setup.

I'd prefer seeing the QoS page under saturation instead of the bandwidth monitor's graph. Much easier to discern what is going on if you see the actual QoS classes and their utilization rather than what the devices eat total.

From what I understand your actual goal here is to ensure that one of the devices can run a streaming service, otherwise all bandwidth should be evenly split among devices. To achieve that you could setup 2 classes, NORMAL and STREAMING. Make NORMAL the default class, grant each 50%-50% bandwidth and setup rules based on the ports to funnel the streaming bandwidth to the corresponding class. Don't bother with IPs or TCP/UDP for now.

Report back with your findings and a screenshot of the QoS page under saturation.

Bonus questions:
1) Why are you giving the streaming class 60% of your upload? Seems like asking for trouble to me. Does it actually max that class? Which way does the stream go? Are you uploading something or are you trying to ensure that something like YouTube does not buffer?
2) Why do you attempt to differentiate between TCP and UDP in your classes?
3) Do the secondary routers run Gargoyle or have any other method of QoS?

[dsalch]

Sorry for the confusion. Here is a simplified view of the network map.

Code: Select all

                            RouterB ----- Building B LAN/WiFi --PCs
                              |
                              |
                              |
WAN --- Router A  ------  Building A Lan/Wifi ----- PC1
                              |             |----- Other PCs
                              |
                              |
                            RouterC ----- Building C LAN/Wifi -- PCs

Router A is the gargoyle router in question.
Router B and C are generic routers with no QOS operating
Router B and C connect to the LAN/WiFi of router A
There are multiple devices behind router B and C
Router B and C run NAT, using router A as their WAN
PC1 occasionally runs video streaming, requires 3mbps down - 1mbps up
The video stream is a combination of TCP/UDP on specific ports
Other applications run on PC1 that should not be included in the video streaming BW
ISP feed through the WAN is 30mbps down, 2mbps up
Effective BW from ISP is 27mbps down, 1800mbps up

The goals are:
1. Preserve the bandwidth for PC1 at all costs while video streaming. this is the top priority
2. balance remaining bandwidth between building A, B, and C equally, not between devices, but between buildings, all the devices in building a counting against building A total.
3. any building can utilize the full unused bandwidth
4. minimum BW to any building is always 6mbps

[dsalch]

I'll try to address the questions below individually


As far as the main router is concerned, you have 2 additional routers (ROUTER1, ROUTER2) and a PC1? Yet then you say you have another class locally which is a conglomeration of devices on your local (wifi) network? So which one is it?

See the network diagram. It is both, the LAN/WIFI on gargoyle router has a wide set of devices, the other two routers connect to gargoyle lan

How many devices are there?

there are 3 subnets, each has a collection of devices. perhaps a half dozen each.

Which ones actually need special treatment?

A single PC needs special treatment, or rather an application on a PC, and then each of the subnets needs to be shared equally


What are the bandwidth requirements of that special treatment?

3mbps down, 1mbps up when in operation


Bonus questions:
1) Why are you giving the streaming class 60% of your upload? Seems like asking for trouble to me. Does it actually max that class? Which way does the stream go? Are you uploading something or are you trying to ensure that something like YouTube does not buffer?

This is a live video stream, bidirectional, a combination of TCP and UDP

2) Why do you attempt to differentiate between TCP and UDP in your classes?

The application that needs special treatment involves both TCP and UDP streams

3) Do the secondary routers run Gargoyle or have any other method of QoS?
not yet, but I'd like to when everything else is stable. That would allow better balancing inside of each building (behind each router). But one step at a time.

[Daeron]

Assign static IPs to devices using DHCP:
PC1, RouterB, RouterC

Configure QoS Download with classes:
[bandwidth%, class name, match criteria]
25%, STREAMING, IP of PC1 and ports used by streaming
25%, BUILDINGB, IP of RouterB
25%, BUILDINGC, IP of RouterC
25%, BUILDINGA, <default service class>

If you have 30000 set as total bandwidth, this will give each class 30 * 0.25 = 7500 kbps if the link is perfectly saturated by all of them at once (absolute worst case scenario). You could go as low as 24000 and 6000 kbps would still be ensured. All the unused bandwidth is recycled from each class, so without streaming each building would get more like 8000-10000 kbps (33% of total).

I suggest doing the same on the upload side before trying anything else. If you find the stream is really using 1000 kbps (my guess is it won't), you might want to adjust the percentages. If the total bandwidth is 2000, you'd need to set it to 50% to ensure 1000 kbps to that class.

[dsalch]

Thats pretty much exactly where i ended up except for finer tuned percentages on the classes. I do in fact need a minimum of 1000 kbps for HD streaming video. 1500 would be better, but I'm settling due to the limitations of DSL.

Everything is working as long as it stays in saturation and I do not use MIN BW settings. That was my original question, for some reason MinBW on upload breaks it. In theory it shouldnt, right? The only difference I see is that min setting takes bandwidth "off the top" before any class separation, and the rest is shared according to classes. It doesnt seem to be working though. When i set a min BW the other classes loose their allocated percentage.

In a separate thread, I should share the entire network map. Quite interesting. A combination of WiFi links (ubiquiti) and 2 gargoyle routers running off 2 DSL lines. Using ACC balanced between the two routers to share an oversubscribed wifi link. Yea, i know, that makes no sense, and is not relative to this discussion, but I think you guys would find it interesting what this stuff can do I love the new QOS and ACC actions.

[pbix]

Effective BW from ISP is 27mbps down, 1800mbps up

This statements means that the upload bandwidth on the QoS page should be set to not more than 0.95*1800=1700kbps. You should start working with 1300kbps. This mistake alone will break your upload QoS.

I agree with Daeron that setting your download to 24Mbps is a great way to simplify for now and avoid a host of issues until things are sorted. Once it all is working we can increase it back.

I requested a printout of your "qos_gargoyle restart" because I want to see that the DSL accounting statistics are set properly. Yours is an usual DSL setup with very wide asymmetry. I think only the latest version of Gargoyle has a chance to correctly configure. These settings will be critical for your success which is why I requested this. This is why I asked what version you are using.

There is no problem with MinBW in Gargoyle in general. It works fine for me and many other people. We are trying to help you figure out what might be going on in your special case.

BTW, what router model are you using? CPU saturation will also completely break QoS and is a common issue. Your speed of 30Mbps means only a few models can be applied. You can see the CPU load on the status page. These numbers should stay below 0.9 or trouble is at hand.

[dsalch]

My apologies.. I neglected to finish these tasks. Been busy in my world.

I do agree, one of the most tempting problems using QOS is to eek out every drop of bandwidth, especially when things are tight. I probably went too high on upload BW. It seems to be working at 1700, but perhaps not reliably. I will adjust down to and test.

I will get the QOS stats as soon as I have a chance to have test traffic.

The router is: TP-Link TL-WR1043N/ND v2

And yes, I have seen it climb as high as 2.x on the cpu rating under heavy load of all three classes. This may well account for the related problem of QOS load above QOS limit. I also suspect there may be some correlation between QOS breaking and load levels. Perhaps this explains why my min BW setting "breaks" QOS intermittently.

I do have a PC based router I was planning on inserting for QOS that seems much more capable. Just need to find a third network card that will work in the old PC.

AS for loading itself, Now that the setup is basically working, QOS Link Limit drifts down to about 20,000kbps in "normal" usage from the 30,000 max. So that is working now I watched it for quite a while yesterday under various conditions and it is tracking nicely, right at the edge where the bandwidth charts flatline across the top. There were a few times when QOS quit (one class swamping BW from another) in the area just before flat line (around 18000kbps total).

There seems an area where the load is just under the limit, and the limit is rising but QOS is not fully applied and therefore bandwidth not shared. Once the link limit rises to the max, it starts sharing according to percentage, but as link limit is climbing under ACC the sharing is not always proper. Need to watch this more to understand.

Thanks!