I think this is the section of the Wiki that applies here.
An important fact about fairness is that it has a cost. In the case of QoS the cost will come in terms of reduced utilization of your WAN link. Lots of work has been invested in making this cost as low as possible but you are going to take a 5-10% hit on your WAN throughput to get fairness. This is the cost of QoS and if you do not want to pay it you can stop reading now. When you are having the problems that QoS can solve you will be more than willing to give up 5% of your bandwidth to solve them.
It means QoS, when active, always has a cost in bandwidth. And you will pay the highest cost when MinRTT is active. The costs mentioned are only approximate and vary in each case. You will have to note your own costs as you have been doing.
Another important sentence for you from the Wiki,
The lesson here is that if the WAN is not saturated your QoS setup will not matter much, all packets get immediately transmitted.
This means that unless the link is operating at the link limit shown in the ACC status section your QoS setup is inactive and doing nothing. You need to wrap your head around this fact because I can tell you think otherwise. As you noted link limit climbs slowly and falls rapidly. The only way to make it move in either direction is to saturate the link which is when QoS becomes active. It does not reset unless QoS is reinitialized.
The above conflicts with your sentence:
Fa55u wrote:Although it specifies only when the class is active which doesn't explain why I see slower speeds when no class is active.
If no class is active then you are seeing no speed because there is no requirement by anyone for any data. Again QoS is doing nothing in this case and not responsible for any phenomena you think you are seeing.
Concentrate on the case when your link is saturated. This is the only time QoS is doing anything. And keep studying because your learning
