uTorrent doesn't like 4Mb pieces or what?

[Firon]

Lexxington: a peer has to send an entire 16kB block or it gets discarded, hence why sending only 1 byte won't work

[SigourneyBeaver]

Is there any remote chance this has to do with having a fat32 partition instead of NTFS?

I only ask because I know 4BG is sort of a cutoff point for fat32 before it messes up a file.

[Martin Levac]

In my case, it can't be since I use Win2k with NTFS. I don't know about Lexx but he did say the problem went away after the bad peers were taken out. I believe him because he gave me the torrent for me to try and I did get several bad blocks from multiple peers in a very short time, some of them not in the list he also sent me.

My conclusion was that uTorrent did not have a problem with large pieces because after the bad peers got cut, no other problem surfaced. Because the bad peers were spread over many domains, we thought that it wasn't a combined effort to poison the torrent but the facts still say multiple peers were poisoning the torrent. When it happens to me, I just look for a better torrent otherwise I can't do anything but waste my bandwidth and time until uTorrent gets the job done. Getting rid of bad peers at 20MB a pop is quite a job indeed. It is possible that a single peer can send multiple bad blocks for the same piece so that could accelerate the process if it sent 5 bad blocks of the same piece. Then it would be cut after just one 4MB piece. But that's being optimistic.

ML

[Switeck]

Lexxington: a peer has to send an entire 16kB block or it gets discarded, hence why sending only 1 byte won't work

Yes, so it can corrupt 1 bit or byte out of 16 KB...and take 1 minute to trickle that 16 KB to you. The end result is still corruption and it doesn't cost them a lot of bandwidth to do it.

Yes, adding the IPs of the corruptors to a separate PG2 block list does indeed enable the torrent to procede without any further hassle.

Unfortunately, this doesn't help anyone else in the swarm.

I disagree. If the corrupters are bad, then you can partially counteract them by banning them from you and putting your upload slots low(er) so you quickly upload whole 4 MB chunks to people. Less chance of corruption if you upload quicker. Then the faster peers/seeds in the swarm will latch into you and upload to you alot. More people "win" that way.

[Lexxington]

Lexxington: a peer has to send an entire 16kB block or it gets discarded, hence why sending only 1 byte won't work

Ahh, I see what you mean now, but I think Switeck already knew that.

@Martin,

re 5 bad blocks of the same piece. Unfortunately this won't speed up uTorrents hash fail banning function, because it's only after the whole piece is downloaded that a SHA-1 hash check can be performed. If it fails, then you can't tell which individual 16kB block(s) failed as they're not hash checked separately.

It's even more complicated if more than one peer was involved in downloading the same piece. All you can really do there is tag all of them as possible suspects. I believe (from observing it) that that's what uTorrent actually does?

I can envisage a fairly good counter corruption algorithm for that problem though.

Say for example there's 5 peers involved in a corrupted piece. We don't know who exactly is innocent or guilty. Imagine the piece as a row.

OK, we take each of those peers separately and assign them exclusively to a separate piece each. We now have a virtual column forming a 'suspect matrix'.

When those pieces are complete the hash checks on the completed pieces will eliminate ALL the non-corruptors from the 'suspects' list.

Any hash fails from this 'column' will now not only reveal the culprit(s) but in addition has given them two chances to get it right. Any hash fails here IMHO should therefore result in an INSTANT ban for the peer involved.

This is MUCH more efficient than a wait until 5 hash fails as employed currently.

What do you think?

Incidentally, BitComet is absolutely no better in this respect. It fails to correct or ban these peers as badly as uTorrent does. It's just better at downloading the data (corrupt or not). LOL

I've got ideas also to counter the non-hash checked blocks too as an addition to the BT protocol, but don't know where to suggest the additions?

@Switeck,

Yes, good points there m8. I'd forgotten about the time wasted in uploading back to the corrupting clients.

Incidentally, I wonder if this point further emphasises that the corruptors are not organised. None of them were seeds. All of them had varying amounts of completetion from 16-85% or whatever. They may be completely unaware that they're screwing things up.

However (and I think this is important) if they were banned by ALL the clients involved in the swarm, they must SURELY get the message that something is wrong forcing them into looking for solutions here and elsewhere?

@SigourneyBeaver (nice name)

Like Martin I use NTFS but with WinXP. The 4GB limit only applies to an individual file in FAT32 though. The files in this torrent are all around 350 MB or so.

[Xu]

Lexxington:

The Bittorrent protocol does quite well on its own. Instead of proposing additonal inblock checking, you could simply use a smaller piecesize. Bittorrent allows from 32k up to 4069k?

The real problem here is 4MB pieces and a public tracker, which makes the torrent very easy to corrupt and the corruptors hard to find. Obviously, when creating a torrent, there are tradeoffs when choosing your piece size. On a trusted network, 4Mb would be probably be fine for this torrent. On an untrusted network, 4Mb pieces make poisoning very easy.

[Lexxington]

Lexxington:

The Bittorrent protocol does quite well on its own. Instead of proposing additonal inblock checking, you could simply use a smaller piecesize. Bittorrent allows from 32k up to 4069k?

The real problem here is 4MB pieces and a public tracker, which makes the torrent very easy to corrupt and the corruptors hard to find. Obviously, when creating a torrent, there are tradeoffs when choosing your piece size. On a trusted network, 4Mb would be probably be fine for this torrent. On an untrusted network, 4Mb pieces make poisoning very easy.

Yes, I agree about the piece size. The recommended piece size for torrents up to 8GB or so is 512kB according to wiki.theory.org. The trade-off of course is the larger torrent file size. Not so much a problem these days maybe though.

That said, I think the public tracker and DHT are more beneficial than detrimental because of the larger peer base.

I don't think it hurts to improve what we have already and make the weaknesses in the BT protocol less exploitable. The same thing had to be done with eMule and eDonkey.

Trust does seem to be the key to this. Unfortunately, BT relies too much on trust when the piece size is larger than 32kB.

[kurahashi]

In my case, it can't be since I use Win2k with NTFS. I don't know about Lexx but he did say the problem went away after the bad peers were taken out.

Well, _you_ are using NTFS partitions, but how about those peers which are sending the corrupted data? Have anybody checked clients behaviour when trying to download >4GB files on FAT32 partition?

[Lexxington]

In my case' date=' it can't be since I use Win2k with NTFS. I don't know about Lexx but he did say the problem went away after the bad peers were taken out.[/quote']

Well, _you_ are using NTFS partitions, but how about those peers which are sending the corrupted data? Have anybody checked clients behaviour when trying to download >4GB files on FAT32 partition?

The files in this torrent are all around 350 MB or so.

[altie]

Lexx, first off let me congratulate you on finding out what was plaguing your torrent. Martin Levac, you did a good job helping out. Lexx I noticed that you said you were trying to download a Star Trek torrent. In which case, I assume you're a Trekkie and for that, I'm glad to have met you as well. I am a Trekkie too. I have to say though, that for a Trekkie some of your replies to some of the posts here have been appalling. You seem to act like BitComet, preferring only to talk to people who cater to your whims and fancies. Anybody who doesn't see your way, gets a fat insult in the face and you pack them off to your "people who don't deserve to be in my presence" list.

I'm sorry to have to break it to you like that. But then again, I'm expecting to be insulted as well soon enough, only because I don't side you, right? I hope, against all hope, that I'm wrong. Please, prove me wrong.

Other than that, please have a pleasant day. Thanks for reading my post.