Overhead over 10% of upload speed?

[debigG]

Unlike many people here I am not experiencing any problems with 2.0.1 in overall speed but just have a question about the overhead.

I am currently uploading at around 900kbs - 1mbs, and am experiencing an upload overhead of over 100kbs. It stays around 120-130kbs. I though I read somewhere that the overhead was supposed to stay at around 5-10 kbs, so that's why I am asking. Is the overhead so large because I am uploading at high speed? or is something wrong with it?

About my downloads:

currently download a very very slow torrent (was slow before 2.0.1 so its got nothing to do with that) but my download speed is around 2.5Kbs yet my overhead is 25kbs. This is huge compared to my download speed, but again does the download overhead have anything to do with seedings? Since I am seeding around 35 torrents at an overall of 1Mbs, this may affect the download overhead as well?

Other than that, experiencing no problems with my speeds (I think it is because both my upload and download are set to unlimited with no cap whatsoever).

Anyway if somebody could clarify my questions on the overhead it would be appreciated

thank you

[rafi]

to better answer that - post a screenshot of your network overhead graph.

[debigG]

Today still the same problem as yesterday. Here's a screen shot.

Although hard to see, looks like the retransmission upload and header upload are taking up most of the overhead. Not sure what these are; clarification would be nice.

thank you

[rafi]

yeah, seems that upload retransmission is most of it. Here is a similar test I did at the time.

My conclusion was that some ISPs are messing up with UDP packets. I think there are 2 possible solutions:

1) set an upload speed limit just bellow your connection cap (till your retransmits drops, hopefully).

2) revert to TCP ( tip B.2 here )

[Firon]

Hmm. I wonder if that is merely mislabeled or very real upload overhead being generated. I'll investigate this, since it really shouldn't be generating almost 100 kbyte/s in overhead from retransmissions (!).

[Ther0]

I have the same problem, i have massive overhead in upload (almost all the bandwith dedicated) and a lot in download.

Here 2 screenshoot.

[rafi]

I bet it's NOT "mislabeled". When this occurs uT should:

1. throttle back (enabled by a user option ?) .

2. auto-speed limit per user pre-defined connection caps (-X% as tested in the setup guide ?)

3. better manage uTP traffic ...

[curkcuspy]

My upload total seems to run 50 to 100% higher than my payload.. that strikes me as a little high. Although It actually appears I'm not uploading anything and it's only overhead despite the numbers.. if only there was a way to reduce the scale of this graph cause when i'm seeding my 1000 torrents, the upload doesn't usually get above 10k per second. I know my ISP is brutal for throttling, but don't know what kind of effect that would have, or if there's anything I can do to help..

[Greg Hazel]

I bet it's NOT "mislabeled". When this occurs uT should:

1. throttle back (enabled by a user option ?) .

uTP already "throttles back" if retransmissions are required and loss occurs. User option for throttling back (or not) are a great way to allow people to screw up the internet for others if they decide their traffic is more important.

3. better manage uTP traffic ...

Better manage how? Someone is dropping packets, and it's not uTorrent. uTP deals with loss the same way (a little more conservatively, actually) as TCP.

[rafi]

Greg Hazel:

uTP already "throttles back..."

after/for how long ? it surely doesn't look that way on our graphs. If it *should* be doing that - maybe there is some bug preventing it ?

... deals with loss the same way ... as TCP

showing some test procedure/results (+recommended tools you've used for those) would be nice (and much better help for repro them than unofficial/unprofessional measurements some of us do)

Better manage how ...

... there was the #2 idea too: maybe - anticipate it

[Greg Hazel]

Greg Hazel:

uTP already "throttles back..."

after/for how long ? it surely doesn't look that way on our graphs. If it *should* be doing that - maybe there is some bug preventing it ?

There is no "surely" about it.

Your graphs do not show enough information about why the retransmissions are occurring or whether TCP is also experiencing retransmissions and at what rate (uTP does not have that information to display to you, since Windows is responsible for it).

To answer the question of after/for how long you need to understand how TCP works. There are two interesting components to "after when": On loss, where later packets arrived successfully, TCP retransmits and reduces the congestion window size to 50% of the current size. uTP reacts long before loss, based on latency, but when loss does occur it reduces the congestion window size to 78% of the current size. On timeout, when no data was ACKed within the RTO, TCP resets the congestion window size to 1 packet and send that one packet. uTP behaves the exact same way.

The answer to "for how long" has to do with the maximum congestion window increase per RTT. TCP uses one packet per RTT *regardless of latency*. uTP uses roughly two packets *maximum* if there is *absolutely no latency*.

So, if you had a super fast network, and were randomly dropping packets, uTP and TCP would both retransmit quite a bit, however I suspect uTP would achieve a slightly greater throughput, since there would be no correlation between latency and packet loss. In this hypothetical situation, and probably many others, retransmissions would not be "overhead", since the original data was not really sent at all.

... deals with loss the same way ... as TCP

showing some test procedure/results (+recommended tools you've used for those) would be nice (and much better help for repro them than unofficial/unprofessional measurements some of us do)

For uTP testing, we run the uTP code directly. You however are welcome to write NDIS drivers which drop packets to your liking to observe the behavior, and a Wireshark packet parser which displays the uTP packet information. The header is documented here: http://bittorrent.org/beps/bep_0029.html#header-format

If you would like to get to the bottom of any retransmissions you are seeing in the wild, I suggest doing an apples-to-apples comparison of one data flow with uTP and TCP, collecting Wireshark information for both tests, and analyzing the retransmission rates in both, and the size of the window at the time of retransmission.

[rafi]

Firon:

I'll investigate this, since it really shouldn't be generating almost 100 kbyte/s in overhead from retransmissions (!)

Well, my investigation (using start-of-the-art tools like Netmeter + Netlimiter @20K UL limit) shows that those retransmits graphs/data cannot be trusted, and might be bogus calculations and display since the real measurements are completely different. I suggest you/devs try to compare your data to other tools too, to confirm, and fix it...

uTP/TCP apple-to-apple view ... (x/y = uT/netmeter data)

Edit:

Just keep in mind that NetLimiter can have effect on accurate measurements, so it really has to be further investigated....

Greg:

...apples-to-apples comparison of one data flow with uTP and TCP...

Done/posted plenty of such comparisons already. Still, uT does NOT act here upon those pseudo-re-transmits (if they are not just GUI bug), and *does not* throttle back. And I think I'll wait till you are done exploring my previous data/capture that you asked for. Do your own research/Wiresharking on this one. For me, I have my limits set properly, so not to cause any ISP issues/retransmits