Read an XML ISP usage meter and slow / halt traffic accordingly

[mjd26]

My concept for a feature is something for those users who are on restrictive ISP plans which measure uploads and downloads and don't give unlimited throughput (as I would wager 90% of torrent users would be).

The idea is that almost all major ISPs around the world will (if you look hard enough) have an XML data sheet or some other publicly accessible table of how much of your quota(s) you have used.

It may be presented with downloads, uploads, local traffic etc. and should include the overall limit as well as the amount used.

I figure it shouldn't be *too* difficult to create a system whereby uTorrent can poll that file at a configurable interval for configurable values and act in a configurable way.

Eg.

In my case I have a 40GB download limit and a 40GB independant upload limit.

In my ISPs XML data sheet the variables are represented as

upload-used

download-used &

quota

I have a 40GB quota for both upload and download (independantly) so uTorrent could accurately identify my % quota based on the data.

I would want it to then slow my upload rate accordingly to prevent me getting shaped by my ISP. So as I reached (eg) 80% of my quota it could drop my speed to 40% of standard. Then when it reachs 90% of quota it could drop again to 20% of standard and continue to cut it until it will prevent all uploads if I am too close to my quota and risking shaping. The same could (optionally) be applied to downloads.

I figure that the user should provide:

the url for the xml / data feed.

the variable names which the system should look for

The client app should then have options on how you want it to act.

You should be able to define whether it acts on % of quota used or specific values being exceeded etc. etc.

Peoples thoughts?

[Switeck]

Definitely limit your upload speed to about 50% or less.

Hopefully your download speed won't overrun max without limits.

[mjd26]

"Definitely limit your upload speed to about 50% or less.

Hopefully your download speed won't overrun max without limits."

Why would I want to do that?

I want to let people download whatever they want from me as fast as possible for as long as possible.

I'm fairly good at managing it so that it won't flood my possible sessions and crash my modem and things like that.

The only problem I have is potentially blowing my cap with my uploads.

I do not want to artificially slow it (beyond what is obviously necessary to prevent the connection getting flooded) unless I have to.

[silverfire]

If your router supports it, use SNMP and MRTG//RRDTool. That's a much better way of measuring bandwidth. There are also Perl scripts out there that can take SNMP data to measure aggregate bandwidth usage.

[mjd26]

I don't want to measure the usage on my router.

My ISP is measuring the usage and I need to go by what their usage meter says, not what I calculate as my traffic because the two WILL differ and my ISP will ignore me if I tell them that my usage has been calculated differently to theirs.

Plus, I don't want to control the bandwidth on the router, that will invariably be done by dropping packets and causing errors and things like that. I want the software to control the bandwidth itself based on usages which have already been measured by someone else.

Yes, I could measure my usages myself using my router, but that is likely to be inaccurate.

I could also control speed through my router but it is likely to cause errors in the users connecting to me and also problems with other programs.

It would be much more efficient to just have the client software poll for usages and then change variables.

The variables you're setting already exist (upload speed can already be manually configured) the data you're using to define the results already exists (the ISP creates it).

[mjd26]

Here's an example of one such XML sheet:

<Account>

<MegabyteQuota>40000</MegabyteQuota>

<QuotaStartDate>2006-10-18</QuotaStartDate>

<PlanType>AdamDirect</PlanType>

<PlanSpeed>24000/1000</PlanSpeed>

<SubAccount>

<UserName>mjd26</UserName>

<MegabytesDownloadedTotal>30555</MegabytesDownloadedTotal>

<MegabytesDownloadedExternal>22917</MegabytesDownloadedExternal>

<MegabytesUploadedTotal>34012</MegabytesUploadedTotal>

<LastUpdate>2006-11-11T17:17:44+10:30</LastUpdate>

<NextUpdateEstimate>2006-11-11T19:04:28+10:30</NextUpdateEstimate>

</SubAccount>

</Account>

The things that I want to use are:

MegabyteQuota (how much you can download / upload per month)

QuotaStartDate (when you started that quota; add one month to figure out when it finishes)

MegabytesUploadedTotal (how many meg out of your quota you have uploaded)

This information, being controlled by the ISP means that you don't have to do any calculations or counting to get to it. Yes you certainly can count it with your router, but you would be hard pressed to find a router smart enough to know exactly when your ISP has reset your quota and to match exactly with what the ISP has counted.

With only those three values you could calculate almost anything you wanted in order to define your upload speed.

I'll use mine as the model for just a couple of potential calculations.

I have an actual upload speed of 1300kbps. So I would normally start with a static upload speed predefined at about 100 - 120 KBps.

Problem is, at 110KBps if I'm seeding a lot and it is kept constant at 110KBps, my 40GB will be gone in 4.6(ish) days and I'll spend the next 26 days uploading at 64kbps.

So I *could* assume that my connection will be maxed out at all times and set a upload rate that would run tight to the end of the month, but that would be setting my upload rate to a whopping 15KBps.

The truth of the matter is that my connection will sometimes be used a lot and other times it will barely be used at all so I need to see just how much quota I *really* have left and then base the speed on that.

EG. (This isn't a model I particularly like, but it's one of the options):

Calculate the percentage of quota remaining:

(MegabyteQuota - megabytesuploadedtotal) / megabytequota

result * max speed = new speed

It's nice and linear, it'll scale itself down in a fairly straight smooth line as your quota gets used.

Or better still: if (megabytequota/2 > megabytesuploadedtotal) then new speed = max speed

then revert to the top one. You eat 50% of your quota at full speed and then you commence slow down based on what's left.

Essentially there are hundreds of different combinations you could throw at it to get effective smooth slowing and prevent yourself getting shaped.

Heck, if you really wanted you could make it as simple as "if uploaded = quota*.95 then stop!".

I won't deny that you'd get more control over it if you did it at the router, but this would cater for all the people who have routers that can't do that sort of thing (which I'd say is most people) and all those people who don't know how (even more people) and it wouldn't require manual configuration because it would use the ISPs data which would reset itself every month (your router could not be relied on to do that unless you provided it with the correct rollover dates which I don't think any of them allow).

[Switeck]

I said: "Definitely limit your upload speed to about 50% or less.

Hopefully your download speed won't overrun max without limits."

mjd26:

"Why would I want to do that?

I want to let people download whatever they want from me as fast as possible for as long as possible."

If you're anywhere near your upload max, you're paying for bandwidth you can't even use!

Your ISP will count your upload overloads towards your limit...then won't even send it out!

Also, 40 GB upload a month isn't really all that good. For a 30 day month, that's 1.333 GB a day. That's 1365 MB, or 1,398,101 KB. That's alot...but it's only 16.2 KB/sec. Say you only run uploads 8 hours a day -- that's 48.5 KB/sec.

Surely you can see that instead of having to stretch out your limited quotas, you just reduce the upload rate and it all balances. If you're wanting to keep torrents alive, it's not so important to upload super-quick...as to be still uploading a week or more later.

[mjd26]

Or (if possible) I could try to get more seeders into the field early (I go super fast early in the month, get a few other seeders out there) then once they're ready to go I can pull back my upload speed because there are other people to share the load with me.

Perhaps this is most important seeing as I am often a first seeder (or close to) for the community that I participate in. So if I do first seed of a 4GB torrent with an overall limitted bandwidth of 40KBps, I'm still seeding even just one or two more things which use 10 - 20KBps up and down, that's an effective download speed for my peers of between 20 & 30KBps. (and likely less)

That 4GB torrent is going to take (Assuming a 25KBps average) 2 full days to get the very first seeder other seeder into the field, if they are the only one to be leeching from me. If there are 2 people leeching from me, double it (plus a teeny bit more for overheads). If, god forbid it should actually be popular and there are 4 people leeching it straight away, you're looking at 8 days to just get one seeder in the field.

If I limit upload slots to 1 or 2 to try and get that early seed spread out there, then it will put more load on those two peers who (it is likely, given that I'm australian and broadband here is really bad) will have a significantly lower upload speed than me.

I think my main premis for using more upload speed early is for the purpose of first-seeding. To get the others out there ASAP and start properly sharing the load.

But still, a mechanism to automatically slow / prevent uploads when you're nearing your quota would be very nice.

[splintax]

µTorrent probably isn't the only program you use to download "stuff". I'd say a far better solution would be to have such support built into a global rate-limiter - such as the router-based solution suggested above, or a plugin for NetLimiter, cFosSpeed, or whichever one of those programs everyone's using these days.

[Switeck]

Me: "If you're wanting to keep torrents alive, it's not so important to upload super-quick"

versis

You: "I think my main premis for using more upload speed early is for the purpose of first-seeding. To get the others out there ASAP and start properly sharing the load."

Is pretty much apples and oranges. We're looking at 2 opposite conditions: initially seeding a torrent versis maintaining someone else's torrent weeks/months later. What I don't quite understand is how having 4 leechers early on means you have to upload 4 times as much to see the first seed. Even without initial seed mode, you shouldn't need to be sending out full copies of the torrent to each leecher. At worst, I wouldn't except 4 leechers to take more than 2 times your torrent's filesize to get all 4 finished. Admittedly, only 1 or 2 may stick around...but that doesn't change much if you seed quickly. ...at least not on public torrents!

I am in the USA and I have ~360-380 kilobits/sec useable upload bandwidth. 42 KB/sec is the fastest I can sustain an upload. I have exceeded 80 GB uploaded in a month by running very close to 24/7 at >30 KB/sec speeds.

[mjd26]

"What I don't quite understand is how having 4 leechers early on means you have to upload 4 times as much to see the first seed. Even without initial seed mode, you shouldn't need to be sending out full copies of the torrent to each leecher. At worst, I wouldn't except 4 leechers to take more than 2 times your torrent's filesize to get all 4 finished."

It's not the amount of upload, it's the rate.

In order to have one other seeder out there you need to have uploaded a total of 1 copy of your file (theoretically, if the peering is working perfectly)

If 4 people are leeching off you the software will (theoretically) split your bandwidth evenly between the 4 of them.

For that reason the speed will be quatered and the time taken will be 4times greater.

It's not that you're uploading more it's that the first copy will be uploaded slower due to splitting it.

Yes, you'll end up with effectively a full copy out there across a few people earlier, but you won't have an actual seed and your speed will still be split.

[Switeck]

If they can't assemble a full copy from each other in that time with you effectively uploading at 1/4 rate of 30-40ish KB/sec, then you're probably dealing with a bunch of TRUE leechers anyway...in which case probably within the hour of one becomming a seed, it'll probably be GONE! ...and you'll be needing to seed another full copy out! Hit-and-running goes hand-in-hand with intentional very slow upload speed or running too many torrents at once which has the same appearance. With too many torrents at once, the user feels they "have" to stop finished torrents because the others aren't getting enough (download) speed.

By splitting your seeding between 2-4 people (or more), the odds of 1 having good seeding habits goes up.