Suggestion: Quota improvement

[monoufo]

Hello. I have some ideas. Our situation is this. Comcast limits our total usage per month. We have more than 10 people sharing this connection. If we let people run wild, we would go past the quota. So we are using Gargoyle to cap internet access to less than Comcast' cap. But there is the issue of fairness.

If there was just the single quota, then the heavy users would use it all and block out the light users. We can't let this happen. So we give everyone an individual quota. The multiple IPs per quota feature made this easier to manage, as many people have more than one device. We set up static IPs for everyone.

The problem with this system is that all the internet does not get used up. The heavy users run out of internet before the month is up and beg for some more. We often give people more internet, because the light users won't be needing their whole share. This creates an undue burden on me too. I want a way for the heavy users to use up the unused internet, but not be able to block the light users from access. I want the heavy users to have the right to use their quota up however they want. It is on a monthly basis. I don't want to restrict by the day or by the week. I also want to give the light users the ability to chose to be heavy users any time they want to.

To be fair, everyone needs to be guaranteed a certain minimum amount of bandwidth. But to also be fair, anything not used should be able to be used. Those of us who want more internet shouldn't have to see it go to waste and disappear at the end of the month.

I would like a more powerful quota system that allowed me to come up with some more advanced rules to cover this situation. It is a hassle to manually override the quotas and causes unfairness.

maybe a quota system that knew the date would help. Maybe quotas could automatically be risen a couple days before the end of the month. Maybe the normal quotas would apply for the first 28 days of the month, and a new quota system would take over for the rest of the month. Maybe it could even know how much is left unused and adjust accordingly. Basically I need some kind of system that could automatically adjust quotas somehow at the end of the month, yet have little risk of actually getting to 100%.

maybe there could be two global quotas, a hard and a soft. The soft would be set a little under the hard quota. Once the soft global quota is reached, anyone who has already hit a certain threshold, would no longer be allowed any internet the rest of the month. Those who have not hit that threshold can continue to use what is left in between the soft and the hard quota. The soft and hard quotas would be active all month long. There would also be individual quotas. The individual quotas could have an expiration date set, like the 28th. This will prevent people from racing to use ununsed internet before the end of the month.

anything you decide to do to make the quota system more powerful could potentially allow us to come up with rules that are a little more close to fair.

Comcast won't let us pay them for more internet, they will not let us upgrade to a business account, since we are already flagged for heavy use. The cost of the internet is basically built into rent, so everyone pays the same price for the internet. I'd be willing to pay a higher share of the price for a higher portion of the internet, but that is just not practical.

[Eric]

The problem here is that it's impossible to design something that will predict how much "light" vs "heavy" users will need. I like to think I'm a reasonable programmer, but I can't program a crystal ball (and if I could, I would probably keep it to myself and make a few well placed investments...).

Here is what I am currently planning to implement (though I'm not sure when it will be ready): I'm going to add the ability to re-classify traffic into a specific QoS class when a quota is reached instead of cutting off traffic entirely. So, you can setup a QoS class for "over limit users", that has strict limits so they get service but service that is much,much slower than what they would otherwise get. If you ever need to cut them off entirely you can adjust the parameters of that class to 0b/s, and then set it back to a slightly higher setting at the beginning of the next month. How does that sound?

[uncle john]

Shaping once allocated bandwidth is reached has been used by all DSL and cable ISPs in Australia for over a decade and is therefore well understood by this user group.
I'm sure this feature will be extremely popular among Australian users and the many other countries that use shaping.
I'd suggest that this proposed feature would be more important to this user group than any other proposed feature (e.g. captive portals etc.).

[monoufo]

Shaping is important for a lot of people, so do that when you can. I don't know if it would help me though. That would let people at least have some internet access if they reach their quota, which is good. It would be mean to cut them off from email when there is still internet available. That doesn't solve the problem of having 250GB of internet total, trying to use up 230GB of it, but only actually using 180GB. That is 50-70GB of internet that goes wasted.

The current solution is to just keep pushing the max individual quota higher and higher each month, manually adjust it for those that beg, and let people use different IP addresses to use up the quota of someone else who isn't around to use theirs. This system in less than ideal and only works while I am in power. If I lose power by upsetting the real administrator of the network, or if I move out of the house, the rest of the residents will be at the mercy of the real administrator. He won't be as easy on them.

If you add calender support to the quota screen, I can come up with something that is less bad, but you are right, there will never be a perfect system.

maybe something like this

User1_quota1 !st-28th 20GB combined
user1_quota2 28th-end 10GB combined

User2_quota1 !st-28th 20GB combined
user2_quota2 28th-end 10GB combined

heavy_quota_combined user1+user2 XGB combined
, where X can be adjusted once a month or so if needed

You have the multiple quota support, it just needs support for the date. Combine that with QOS throttling of the heavy users so they couldn't possibly use more than a a gig or so over a week or two, and i think the system would work well. But even just the date support for now would be good. That should be easy right?

[uncle john]

If some of the householders consistently don't use all of their allocation why not find out whether they'd agree to lower theirs and raise others?
On the other hand you may be surprised how much of an impact a bit of shaping will have on usage patterns.
Why re-invent the wheel? I've seen ISPs here in Australia use all types of schemes to regulate usage over the past 10 years. They try them for a while, sometimes up to a year or more, and then drop them. They all come back to schemes based on shaping.

[monoufo]

I don't want shaping. We pay a lot of money to get the fastest connection they will allow. I want things to be fast for myself, and for everyone else.

The problem with asking people to allow their quota lowered is what happens when they move out of the house? People are allows moving in and out of here. You never know what kind of user you are going to get. I would prefer a system that could be set up by the time I move out, so the remaining admin doesn't have to mess with it all the time.

The only time I want to shape traffic is when people are torrenting so much they slow down video streaming. On occasion, you can't watch videos without buffering.

the real solution to the problem is to get Comcast to give us an unlimited business connection, but I don't think they want to let us have one.

[pbix]

Eric's idea to force classes based on quota seems cool even though its not exactly what monoufo is asking for. There is another post in the General Discussion forum today asking for exactly that. Changing classes in this way is a great way to put users in timeout for exceeding their quotas.

Here's an algorithm for monofo problem I will throw up on the wall. Get your darts ready. The quota system has an overall "budget" limit which it keeps. This limit starts at zero every month and then increments hourly on a rate such that at the end of the month it matches the monthly limit. So this is a ramp function and the data budget.

The quota system also keeps track of the total amount of data transmitted by the router for the month. If the total is below the ramp function then no action is taken. If the total is above the ramp then all users that are above their monthly quota are thrown into a new QoS class (time out corner). Once the ramp again exceeds the total data these users are once again allowed to play.

This algorithm has some fairness in that all uses get a quota amount a month. The sum of all user quotas must be less than your monthly limit, maybe a lot less (or zero) depending on how fair you want to be. To make sure that you do not go over the monthly limit you will have to reduce the ramp function target by the sum of the monthly quotas.

Example:
Comcast limit = 100GB
Total of all user quotas = 20GB
Ramp target = 100-20 = 80GB.

So basically 20GB is fairly allocated and 80GB is a free for all. At the end of the month no more than 20GB is "wasted".

[rfdude]

I like the concept of soft quota approach. My case involves a low security WiFi for an HSPA+ fed, RV park hotspot, where a daily quota (say 300Mb) stops big downloaders, but is reasonable for many. Most traffic would be on weekends. The sudden death quota enforcement is rather harsh. A reduction of BW to dial up like speeds would be more polite. The only purpose of Gargoyle would be to try and prevent overage charges on the HSPA+ data plan (ridiculously low caps... 10 Gb) as well as monitoring usage.

Along with the soft quota, my wishlist would include a simple splash screen to welcome people to the service and to keep their usage reasonable.

Otherwise, I've found Gargoyle easy to set up on an Asus WL-500GP v1, and an easy and elegant GUI.

Any alternative suggestions to accomplish the task above is welcome.

rfdude

[cha]

Here is what I am currently planning to implement (though I'm not sure when it will be ready): I'm going to add the ability to re-classify traffic into a specific QoS class when a quota is reached instead of cutting off traffic entirely.

OK, I'm probably missing something and oversimplifying things, but this, and I'm sure I've read somewhere that I have to get hands dirty if I want to do it myself.

But wouldn't a simple shell script works as a simple workaround? Like adding a new bandwidth limiting QOS class when the quota is reach for the specific IP, and add a line for the quota reset script to remove such rule, maybe using a specific name for the QOS rule?

[uncle john]

... I'm going to add the ability to re-classify traffic into a specific QoS class when a quota is reached instead of cutting off traffic entirely. So, you can setup a QoS class for "over limit users", that has strict limits so they get service but service that is much,much slower than what they would otherwise get. If you ever need to cut them off entirely you can adjust the parameters of that class to 0b/s, and then set it back to a slightly higher setting at the beginning of the next month. How does that sound?

The reason why this approach (throttling) is used by ISPs is because they're not interested in making any judgement on the content their customers want access to. In fact I'd even suggest that increased content per customer is looked upon favourably since overall support costs would go down (ie. one on one phone calls with customers etc.)

On the other hand I'd suggest that parents (and if not parents, then educational and government authorities etc.) are becoming increasing concerned about the adverse effects that overexposure to digital media is having on society in general.

So lets get real. The reason why quota disappears at such a high rate is not because of people reading and contributing to forums such as this. Nor is it because of the hours developers put into programming and testing etc. It is primarily because of the amount of bandwidth people devote to watching and downloading videos.

So now I'm going to make what I suspect will be viewed as a fairly radical suggestion. Instead of having the quota limit trigger a re-classification of "speed" for the user/device involved, let it trigger a Firewall Restriction instead. For example, no more Youtube access. Or no more P2P or Bittorrent until the new period starts.

I'm sure some readers will think this is somewhat drastic. Why not just throttle Bittorrent traffic instead? Think about it. This will not address the real problem of screen time. So what if the number of movies downloaded over a particular period is reduced. People can still come back and watch the stored movies in real time. So throttling doesn't have much impact on time shifted viewing of movies. Nor does throttling Bittorrent affect viewing in real time e.g. Youtube etc.

I look for to seeing the comments of others on this subject. Or if no comments are made it will still be interesting to see whether my comments attract any viewers.