I need an expert's advice! - ... about "wasted data"

[descra]

hi all,

First of all, uTorrent is the best bittorrent client - that's no daubt.

But ther's a one serious problem ...

A number of wasted data could really make you angry!

I read all FAQ (propably 4 times) and I found out that is NOT the uTorrent problem and number of wasted data always depends on every single torrent - for example when torrent is broken, poisoned etc.

ok, I can understand it but ......

Today for some test I downloaded some movie.

Resaults?

uTorrent: movie 700 MB, wasted data 40 MB !

Azureus: the same movie! at the same time tracking on the same tracker, almost the same number of peers'n'seeds, wasted data? - only 5 MB !

In my opinion is a real bull schit that uTorrent has no influence of number of wasted data!

So ....

I need an expert's advice.

Could you tell me whitch settings - I mean "Advance options" and others may affect a number of wasted data?

I'll be really thankfull for your help and some wise advice.

thanks in advance

[Ghost21]

Why would there be any settings to increase or decrease wasted data?

[Klaus_1250]

Could be usefull for people using low bandwidth connections (though they're less likely to have wasted bandwidth) or who have to pay per megabyte.

As far as I know, wasted data occurs when the same data is downloaded from different peers (among others) and I think there are a few switches you can make to prevent this, at the cost of overall speed. But on my installation, I don't see a lot of wasted bandwidth.

[descra]

Why would there be any settings to increase or decrease wasted data?

sorry for being not enough precise

I mean whitch settings may theoreticly influence to downloading less number of wasted data.

I'm realistic and I know that generally it's NOT posibble to download some stuff without even the small number of wasted data but you should admit that 40 MB per 700 MB movie is a "bit" too much, especially when you downloaded the same movie at the same time, using different bittorrent client and the number of wasted data was a lot smaller - like I said before:

the same movie on the same tracker durring very similar number peers'n'seeds

uTorrent: 40 MB w.d

Azureus: 5 MB w.d.

So ...

Could you tell me whitch setting may theoreticly influence on number of wasted data - I mean for example: maybe disk cashe or number of open upload slots or some others Advance Settings ?

please help

[descra]

Here are the list of options witch I think they may have some influence on number of wasted data and I'll be really thankful if someone verify that and tell me where I'm wrong and why (I don't say that they may increase or decrease but I think they may something to do)

1. Disk cache - "Write out untouched blocks every 2 minutes", "Write out finished pieces immediately" and "Remove old blocks from cache "

2. Protocol Encryption - "Allow legacy incoming connections"

3. DHT and peer exchange enabled

4. Too much numbers of trackers, for example 3, 4, 7 and more

5. Report a different IP to the tracker

6. net.max_halfopen

7. diskio.flush_files

8. diskio.smart_hash

9. diskio.coalesce_writes

so ... could you tell me if some of these settings above may have something to do with wasted data?

I mean witch of these options in 100% don't have any influence and why ?

... and witch one theoreticly may have some influence and why ?

... and at last witch one for sure have something to do and of course why ?

thanks in advance for your help

[DreadWingKnight]

1> No effect - Disk cache doesn't affect wether a peer will send you part of a 16k block or all of it.

2> No effect - Encryption has nothing to do with the amount a peer will send you

3> No effect - DHT and Peer exchange have nothing to do with piece data transfer

4> No effect - Trackers have no bearing whatsoever on piece transfer

5> No effect - Reporting a different IP to a tracker will either fuck you over speed-wise or get ignored by the tracker

6> No effect - A half-open connection isn't sending piece data

7-9> No effect - Disk IO controls have nothing to do with piece data transfer

[Switeck]

Setting bt.prio_first_last_piece to true might increase wasted data a tiny amount, as µTorrent would be particularly aggressive at trying to get the first and last piece of a torrent. (or is it 1st and last piece of each file in a torrent too?)

I believe net.low_cpu set to false (default setting) means run aggressively and queue up lots of pieces at once (even duplicates from different peers/seeds?) -- while setting net.low_cpu to true would queue up fewer pieces at once. That in theory might have an effect on wasted data amounts.

How many connections per torrent you allow would certainly have an effect on wasted data. If you only connected to 10 seeds/peers PER torrent, who are you going to get duplicate data from?

Also, how many upload slots you allow (and how fast each uploads on average) might have an effect on how many peers are uploading back to you at once -- if you have too few, the only peers that will consistantly upload to you are the ones you are uploading to. If you have too many, then peers will upload to you very seldom and in random bursts. (optimistic unchoke for the win!) Seeds will likely upload to you regardless, but they often aren't any faster than peers and likewise are pretty random in who they upload to. Having only a few peers uploading to you quickly (because you're only uploading to a few peers quickly, because your upload slots are only 2-5) would seem to reduce the chances of duplicate data. Having LOTS of peers uploading slowly to you at once would seem to cause immense amounts of wasted data.

Even running multiple torrents at once (versis 1 at a time) would increase wasted data considerably, simply because you can't be uploading as quickly per torrent and such peers on each torrent would be uploading back to you more sporadically and slower.

I do know that µTorrent goes into "endgame mode" when there are fewer than 50 pieces of the torrent left to get. When that occurs, it tries more peers at once for the same pieces so it increases the likelyhood of completing the torrent at all. It really doesn't need to on torrents with lots of peers/seeds -- as that's probably where it incurs the highest wasted data as a result!

Assuming it's not something more insiduous like a bug, µTorrent probably is just more aggressive at getting pieces than Azureus -- which results in more wasted data overall.

A NASTY side-effect of high wasted data rates...is it is likely caused by often having multiple peers/seeds uploading each piece. If there is a seed/peer that is poisoning the torrent, they can "hide" behind the other peers/seeds -- and you'd end up kicking/banning lots of harmless peers/seeds that just had the misfortune of contributing a part of the same chunk that failed hash!

[kurahashi]

My advice:

- try decreasing number of upload slots

- try decreasing number of connections

And last but not least make sure that your upload graph is flat. No zigzags, no electrocardiogram, just pure flatness.

BTW: my usual level of wasted data is ca 0.5% (0.5MB wasted per every 100MB downloaded)

[descra]

First of all thank you all for your answers and took this problem seriously without any silly replys from your side

So ... propably the best thing will be when I simply show you my settings I use.

1. Connection: upload 12 Kb/s (my MAX upload bandwitch is 16 Kb/s and upload is independent from download speed),

3 open upload slots, maximum number peers per torrent: 90

2. Advance (most important settings):

net.low_cpu - false

net.max_halfopen - 20 (I've got patched TCP/IP.sys so this number above 8 is fine)

queue.prio_no_seeds - true

bt.scrape_stopped - false

bt.compact_allocation - false

bt.send_have_to_seed - true

bt.connect_speed - 20

bt.prio_first_last_piece - false

bt.allow_same_ip - false

peer.lazy_bitfield - true

peer.disconnect_inactive - true

diskio.flush_files - false

diskio.sparse_files - true

diskio.use_partfile - true

diskio.smat_hash - true

diskio.coalesce_writes - true

3. Disk cache:

Write cache enabled with all options

Read cache disabled (I read that the read cache is not need it when the upload is under 12 Kb/s)

4. DHT disabled (of course I switch it on when the tracker got down but normaly is off)

so... as you see these are my settings

Important thing is I do NOT download broken or poisoned torrents and in 99% tracker works fine.

I'm not behind NAT

Firewall is well configured

so .... should I change some (or even only single one) of this options witch I talk about above?

Like I sad before I downloaded today some 700 MB movie. Torrent wasn't broken, tracker worked fine, settings was like I said above but wasted data was 40 MB! - a bit too much, isn't it?

the most funny thing is that the amount of wasted data still increase with every newer ver. of uTorrent (now I use the newest one)

Azuerus wasted data: ~ a few MB

uTorrent w.d.: ~ 20-40 MB!

so I got laugh when I read in FAQ that wasted data has nothing to do with uTorrent settings

please help

thanks,

[Nefarious]

that u tried the same torrent doesnt mean u are gonna get the same results, u could have ended connection to a peer that was causing the wasted data when in utorrent and then he was not in azureus connections as u cannot be sure u are gonna connect to the same people or if they just left the swarm, i've seen this behavior a few times when first i got 20MB wasted for 1.0GB downloaded, but then after disconnecting and reconnecting to it i got no more than another 2MB wasted for 2more GB downloaded, it's certainly unfair to compare like that unless u are always getting the same results

[descra]

... it's certainly unfair to compare like that unless u are always getting the same results

... but I got the same resaults all the time and that's the problem - a lot MB of wasted data

I wouldn't be talk about it if it will happend sometimes but it happends always

so that's why I need some wise advices

[Switeck]

Try changing these values to this:

maximum number peers per torrent: 30

net.low_cpu=True

net.max_halfopen=6

bt.connect_speed=10

Are you by any chance running multiple torrents at once?

[Klaus_1250]

I'm wondering to what extend waisted data depends on the quality of your internet connection / swarm size / swarm speed. I use µTorrent with quite aggressive settings, but download mostly from smaller swarms and/orslower ones than my connection (250KB/s down max were my connection is 515KB/s) and I see little waisted data. Usually it is about 0.1 to 1%.

I think Kuruhashi post says the most important things. Keep your connections down and make sure you set proper up- and download speeds (pratical ones, not theoretical).

I think connecting to a lot of peers and download at 1KB/s or less from them is a sure way to get waisted bandwidth. (I think that is where my waisted bandwidth comes from).

[jivan]

hi all,

noob here but get something like 6 mb on 800 MB downloads. Although do keep tweaking stuff here & there. One of the things which has not been put out is the ipfilter. So use a good ipfilter.

Other thn that my settings are similar :-

maximum number peers per torrent: 10

net.low_cpu=True

net.max_halfopen=16

bt.connect_speed=25

& my siggy gives my dl/ul ratios. Any suggestions for me anyone :-

[descra]

I'm wondering to what extend waisted data depends on the quality of your internet connection / swarm size / swarm speed. I use µTorrent with quite aggressive settings, but download mostly from smaller swarms and/orslower ones than my connection (250KB/s down max were my connection is 515KB/s) and I see little waisted data. Usually it is about 0.1 to 1%.

I think Kuruhashi post says the most important things. Keep your connections down and make sure you set proper up- and download speeds (pratical ones, not theoretical).

I think connecting to a lot of peers and download at 1KB/s or less from them is a sure way to get waisted bandwidth. (I think that is where my waisted bandwidth comes from).

quality of my connection is more than good - is stable without any gaps or interraptions during downloading or uploading

I think the most important thing is what you said at the end of your post - I mean "connection to lot of peers and download at 1 Kb/s or less from them"

I tried to find out any option in uTorrent settings similar to Azureus one to ignor download speed from peer when is slower than for example 1 Kb/s

I didn't find that and maybe that's the key problem?

I don't know, maybe this is an inner option witch we normally don't notice and haven't access to change it and it's automatic in use but I'll be thankful if someone could admit or deny if uTorrent is able to ignor specific download speed from peer for example if it's slower than 1 Kb/s ?

so ..... is it posible to set in uTorrent settings the minimun speed of download from single peer under witch level that kind of connection will be ignor?

p.s.

When I asked if it's posible that disk cache has something to do with wasted data I got answer that NO

but .... I'm still wondering about one option "Write out untouched blocks every 2 minutes"

Decription of this option says: " ...will control whether µTorrent flushes the chunks from incomplete pieces to disk if the piece has been inactive for 2 minutes"

so ..... theoreticly, is it posible that after 2 minutes uTorrent remove some incomplete pieces only why because they were inactive ??? ..... but in my opinion it doesn't mean that after 2 minutes they couldn't be usefull - maybe after this 2 minutes they could be still ready to use.

for example, some torrent is cuts in 1 MB pieces (NOT parts like *.rar)

I stared to downloading one of this pieces, I downloaded 800 KB (224 KB still to come) but sudenly BREAK - 2 minutes break

and what - that 800 KB segment will be remove ???

even that after 2 minutes could be ready to use and only wait to be finish ?

Situation: I started downloading one of that 1 MB, I downloaded 800 KB but peer from whom I download it sudenly stop sending me that piece

is it posible to finishing downloading that incomplete piece from other peer? ... or it's ONLY posible to finishing downloading it from that first peer from whom we stared ?

thanks a lot,

[Switeck]

Actually, if your maximum number peers per torrent is set to only 10, you don't really need a high halfopen value. The halfopen value and connect speed combined is how fast µTorrent tries new ips to get new peer/seed connections. But if you've set the limit to 10, it doesn't need to be trying 16 half-open at once with 25 new connections (in the handshake process) at a time.

Klaus_1250 was saying if you download from lots of peers at only 1KB/sec or less, you get more wasted data. In translation: if you allow lots of peers (high max connections per torrent) then you increase your wasted data -- as you're downloading from MORE peers at once, at a slower speed each (if you're already downloading at >100 KB/sec), and thus peers are more likely to send you the same thing.

"Write out untouched blocks every 2 minutes" copies the blocks to HDD from ram, it does not then 'forget' where it was at in the block. That's not the problem-causer.

"some torrent is cuts in 1 MB pieces (NOT parts like *.rar)"

Actually, each 1 MB piece of a torrent so set is 1 "chunk" of the torrent -- it is the smallest amount of that torrent that can be individually checked for a bad hash which means that chunk has an error in it.

Once you get 800 KB of a 1 MB chunk from 1 peer, ANY other peer/seed can come along and try to finish that chunk -- and may even be requested at the same time as the original peer is still requested to send the rest! So both can end up sending the remaining 224 KB...giving you 224 KB of wasted data on 1 MB. (21.9% wastage rate!)

You only have indirect control over how many chunks µTorrent tries to download at once. Then again if you put scheduler into seed-only mode, it will finish unfinished chunks (or may only finish current requests) and then only upload what it has. But that would be an absolute pain to test more than once.

The settings we have suggested are meant only to reduce wasted data amounts. Please test them.

Upload speed max is not totally independant of download speed. While downloading, some upload bandwidth is needed to keep the download going. From my findings, it's a 50-to-1 ratio (or worse) -- so if I'm downloading at 50 KB/sec, I need at least 1 KB/sec of upload bandwidth to keep that download going. If I was downloading at 500 KB/sec, I'd need 10 KB/sec or more upload bandwidth...a noticeable amount indeed of my total!

[kurahashi]

Upload speed max is not totally independant of download speed. While downloading, some upload bandwidth is needed to keep the download going. From my findings, it's a 50-to-1 ratio (or worse) -- so if I'm downloading at 50 KB/sec, I need at least 1 KB/sec of upload bandwidth to keep that download going. If I was downloading at 500 KB/sec, I'd need 10 KB/sec or more upload bandwidth...a noticeable amount indeed of my total!

50:1 ratio? Thats really low!

In my case, average ratio is 20:1. That means, every 20kB/s of download generates additional 1kB/s of upload overhead. Additionally, upload generates overhead by itself, in my case it is 8:1 (every 8kB/s of real upload generates additional 1kB/s of overhead)

How does it work in practice:

- my upload limit in uT is set to 24kB/s (192kb/s)

- while seeding only it is using 27-28kB/s

- while downloading at 60kB/s, upload used is 30kB/s

- while downloading at 120kB/s - upload is 33kB/s

- while downloading at 240kB/s - upload is ca 39kB/s - just to be barely safe in the parameters of my day connection (which is real 2048/320)

So, to be able of using my whole 256kB/s download, I have to limit upload in uT from 40kB/s (which I have) almost by a half to 24kB/s. Sounds incredible but its true.

[Klaus_1250]

@ kuruhasi: Are you using SACK and have you set a good MTU size? 20:1 and 8:1 sound too high.

[Switeck]

The 50-to-1 ratio (at best) was for simple file transfers between computers on my network. This lacked BitTorrent protocol overheads and was only a single connection at a time. I have the TCP receive window set to 256960, which also probably lowers overhead...at the price of higher latency due to chained packets. MTU is set to 1500 bytes.

[kurahashi]

Nah, my current ISP provides the net through normal ethernet connection and my settings are ok, sack on mtu 1500, window size = 255*(mtu-40) so...

I even try to play a little with the TCPIP settings lowering mtu a little to 1492 (the only reason to do so is possible communication improvement with the remote peers which are using xDSL PPPoE connections) but I can't see any differences.

[Switeck]

Actually, it probably does take 9 bandwidth bits (or more) to transfer 1 file BYTE. This is due to all the overheads. I even figure it at 10 bandwidth bits per file byte, just to keep the math simpler.

So if someone says they have 300 kilobits/sec of upload bandwidth...I say limit upload speed to no more than 30 KB/sec. It's about the same as 300 / 8 * 0.80

[infused]

I get hardley any wasted data... and I send around 1tb a month. Most clients are uTorrent though, but not all...

[kurahashi]

Switeck, but don't confuse modem overhead with TCPIP and application overheads please. We can distinguish:

modem/physical layer overhead - when your cable or xDSL modem synchronizes, for example 512/128, the real speed available for TCPIP transfers are even 15-20% lower than your synchronization speed (due to PPP encapsulation for example)

TCP/IP overhead - this one is much harder to estimate, usually consist of the all the handshake and ack packets, sometimes IP packets are lost and there is need for retransmission

application (uTorrent here) overhead - all the torrent communication excluding the real data exchange - with tracker, other peers, pex, dht etc.

What we have to remember, is the upload cap setting in uTorrent concerns real data upload speed, without any of this overheads. Which can be really big, as seen this in practice.

[Switeck]

Yes, I know...µTorrent's upload speed cap does not consider overheads...which can add a considerable extra bandwidth burden to the line.

I set 40 KB/sec as my upload max...and it ends up using close to 360 kilobits/sec upload bandwidth -- though that depends on BT traffic.

I did say ALL the overheads, which in µTorrent's case are modem/physical layer overhead, TCP/IP overhead, and application/protocol (uTorrent here) overhead.

And that's assuming failed hashes and lost packets are quite rare.

I'm saying 9-10 bits bandwidth = 1 byte of a file transferred as a rule-of-thumb/rough estimate. But it is good enough for upload speed calculations.

There is certainly a possibility that the Speed Guide's speed test is measuring bandwidth based on file transfer rates, making some of the overheads (at least modem/physical layer overhead) "transparent" in the calculation.

If I recall correctly, PPPoE looks very wasteful.

[Firon]

Chalk up the wasted differences to coincidence.