uTorrent doesn't like 4Mb pieces or what?

[Lexxington]

Hi,

I've been suffering massive corruption on some Star Trek torrents which use 4Mb pieces.

At first I used Bit Comet, then switched to uTorrent but the problem became even worse!

I monitored the peers for hash fails and added the worst offenders to Peer Guardian and the corruption stopped.

The thing is, apart from one peer, ALL the other troublesome corruptors were uTorrent 1.4 clients!!! In addition, most of them were using port 32459 (a default apparently?).

What surprised me was just how many bad peers there were. Over 14.

Is there something in uTorrent that makes corruption more likely in torrents using larger piece sizes, or is it just that the larger piece size itself is a problem?

These torrents are 8Gb or so. Bit Comet works way better than uTorrent does on these.

According to the FAQ here, there's a problem with data mangling on some routers? Anyway, this is a real PITA as it would appear NOT to be deliberate judging by the diversity of the IPs involved?

Any ideas or comments?

[1c3d0g]

I wouldn't be so sure about that. :/ I think it's very well possible that the MPAA, HBO etc. scumbags are purposely poisoning that torrent (and using µTorrent to ease their server load no less!).

[Lexxington]

I wouldn't be so sure about that. :/ I think it's very well possible that the MPAA, HBO etc. scumbags are purposely poisoning that torrent (and using µTorrent to ease their server load no less!). http://img78.imageshack.us/img78/466/mad8yb.png

Yeah, perhaps so, but why then is Peer Guardian not blocking them in the first place?

The Peer Guardian p2p lists are auto-updated every day, but of course someone has to report them for them to get on the lists.

However, I'd be reluctant to report these peers if it turned out that they aren't doing it deliberately. They might be doing it unknowingly. That's really why I'm posting here to see if there might be something askew with the client. There were no Azureus clients at all in the 'hash fail' list.

Then again, what about the other uTorrent 1.4 peers that didn't cause hash fails?

Trying to make sense of it all here!

[1c3d0g]

If the torrent is not poisoned, then we have to take a look at your router and/or PC. Your router may be mangling the data as you said, or your system's RAM may go South soon. :/ Either way I suggest running MemTest86+ or a similar tool to rule out any RAM instabilities.

[Ares]

Are most of the peers (that are sending bad data) from the same domain? If their IP's are similiar, chances are it's being poisoned by a corporation.

-Ares

[Lexxington]

If the torrent is not poisoned, then we have to take a look at your router and/or PC. Your router may be mangling the data as you said, or your system's RAM may go South soon. :/ Either way I suggest running MemTest86+ or a similar tool to rule out any RAM instabilities.

No, that's a pretty bad misquote.

I did NOT say MY router was mangling the data.

There is nothing wrong at my end.

Adding the corrupting peers to Peer Guardian solves the corruption problem.

Please re-read the original post!

You would make a TERRIBLE engineer.

[Lexxington]

Are most of the peers (that are sending bad data) from the same domain? If their IP's are similiar, chances are it's being poisoned by a corporation.

-Ares

Hi,

No, I checked this and there is a very wide variation between the IPs.

Countries and/or ISPs are very different.

[Martin Levac]

Based on all the facts, the problem lies with your machine. Let me explain.

All clients have hash checks so all peers must check their piece before sending it back out. If hash check fails, the piece is discarded by the peer and re-requested and re-sent. If hash check succeeds, the piece is forwarded through the swarm.

As you are discarding one piece that fails a hash check, so do all other peers that come to the same result. No peer can willingly send you a bad piece if that peer has checked the piece's hash with success. Unless the peer is sending you a deliberately corrupt piece. Here's where your machine comes in. You state that multiple IP's of varying locations are sending you different bad pieces and the conclusion is that it can not be a combined attempt by an organization to poison the torrent.

If multiple IP's are sending you bad pieces and no combined attempt is made to poison the torrent but all clients can not willingly send bad pieces because of hash check/discard/re-request, your machine is the culprit. Eliminate all the impossible, whatever is left, however absurd it may be, is the most likely solution.

In fact, you could be a source of bad pieces and I wouldn't be surprised if your IP was banned temporarily by all those peers you send data to.

ML

[Lexxington]

Based on all the facts, the problem lies with your machine. Let me explain.

All clients have hash checks so all peers must check their piece before sending it back out. If hash check fails, the piece is discarded by the peer and re-requested and re-sent. If hash check succeeds, the piece is forwarded through the swarm.

As you are discarding one piece that fails a hash check, so do all other peers that come to the same result. No peer can willingly send you a bad piece if that peer has checked the piece's hash with success. Unless the peer is sending you a deliberately corrupt piece. Here's where your machine comes in. You state that multiple IP's of varying locations are sending you different bad pieces and the conclusion is that it can not be a combined attempt by an organization to poison the torrent.

If multiple IP's are sending you bad pieces and no combined attempt is made to poison the torrent but all clients can not willingly send bad pieces because of hash check/discard/re-request, your machine is the culprit. Eliminate all the impossible, whatever is left, however absurd it may be, is the most likely solution.

In fact, you could be a source of bad pieces and I wouldn't be surprised if your IP was banned temporarily by all those peers you send data to.

ML

No, you obviously do not understand the BT protocol sufficiently and appear to be a fan boy who wants to shift the blame on to something else other than your beloved client?

The fact is that (as already stated) banning the offending peers, results in the torrent proceeding as normal with no further corruption.

The hash fails are caused by received data, not data sent from here!

You do not have to take my word for it either.

I can PM you a link to the torrent in question and you can verify the FACTS for your self if you wish?

[Firon]

Popular torrents are commonly poisoned, and PG2's lists are NOT all inclusive (or perfect/accurate).

[1c3d0g]

If the torrent is not poisoned' date=' then we have to take a look at your router and/or PC. Your router may be mangling the data as you said, or your system's RAM may go South soon. :/ Either way I suggest running MemTest86+ or a similar tool to rule out any RAM instabilities.[/quote']

No, that's a pretty bad misquote.

I did NOT say MY router was mangling the data.

There is nothing wrong at my end.

Adding the corrupting peers to Peer Guardian solves the corruption problem.

Please re-read the original post!

You would make a TERRIBLE engineer.

Thanks for slamming me down while I was trying to give you a hand. Sadly (for you), forum posts such as yours do not influence my perfect mind. And since I've wasted enough of my time with you, you can go figure it out on your own.

[Lexxington]

If the torrent is not poisoned' date=' then we have to take a look at your router and/or PC. Your router may be mangling the data as you said, or your system's RAM may go South soon. :/ Either way I suggest running MemTest86+ or a similar tool to rule out any RAM instabilities.[/quote']

No, that's a pretty bad misquote.

I did NOT say MY router was mangling the data.

There is nothing wrong at my end.

Adding the corrupting peers to Peer Guardian solves the corruption problem.

Please re-read the original post!

You would make a TERRIBLE engineer.

Thanks for slamming me down while I was trying to give you a hand. Sadly (for you), forum posts such as yours do not influence my perfect mind. And since I've wasted enough of my time with you, you can go figure it out on your own.

Don't give up your day job.

[1c3d0g]

I won't.

[Miffo]

Another thread with lexi down the drain

[Animorc]

The fact is that (as already stated) banning the offending peers, results in the torrent proceeding as normal with no further corruption.

You said that yourself. Somehow, that makes me think it was the peers you downloaded from that had errors, and not you.

If that would depend on µTorrent, I can't seem to find any reason why it would work better when you're using other clients. Or are those other clients ignoring µTorrent peers?

[Lexxington]

The fact is that (as already stated) banning the offending peers' date=' results in the torrent proceeding as normal with no further corruption.[/quote']

You said that yourself. Somehow, that makes me think it was the peers you downloaded from that had errors, and not you.

If that would depend on µTorrent, I can't seem to find any reason why it would work better when you're using other clients. Or are those other clients ignoring µTorrent peers?

Well, all the words seemed to be in the right order and they made complete sentences, but they made no sense whatsoever to me!

Would you care to explain just what exactly you meant in those 2 sentences?!

The peers I downloaded from had errors?! WTF does that mean? I didn't?! WTF does that mean?

I said originally that I was first using Bit Comet when the corruption became noticeable. Using uTorrent the corruption was worse still.

In all cases, before and after corruption the d/l speed from the torrent was much better with Bit Comet.

Are those other clients ignoring uTorrent peers?

WTF does that mean?

[Martin Levac]

Lexxington, review the facts and make up your mind.

The bad pieces originate from multiple IP's not based in the same machine/network/domain. This means that it is not an organized attempt at poisoning the torrent like the MPAA would do.

For a piece to be sent back into the swarm, it must have been checked by at least one peer. If hash check fails, the piece is not sent back in. If hash check succeeds, the piece is sent back in. Your client does that, all other clients do that.

Since those peers have checked the hash on all the pieces they sent you, they must be good pieces. It follows that between those peers' connections and yours is a problem that renders these pieces bad. Possible sources of the problem is: Your machine, all nodes between you and those bad peers, all of those bad peers.

That many nodes between you and those peers go bad simultaneously is a very low probability. That many peers have problems that cause piece corruption simultaneously is a very low probability. That your machine has a problem is a much higher probability compared to the other two possibilities. What if the hash info you have stored for that torrent has gone bad for some reason? What could be the reason this hash info has gone bad? I can name a few, system memory, motherboard components gone bad that cause errors not detected or corrected but not properly, connection line noise not properly corrected, etc. All of these problems has happened to me in the past and I know what it does to data and running processes.

PM me the link to that torrent, I'll have a better idea of what's going on with it. I have never had a problem with hash check on my side of the equation and my machine is in perfect health so I'm certain that if there is a problem with that torrent, its source will not be me.

ML

p.s. Lay off the personal attacks.

[Lexxington]

Lexxington, review the facts and make up your mind.

The bad pieces originate from multiple IP's not based in the same machine/network/domain. This means that it is not an organized attempt at poisoning the torrent like the MPAA would do.

For a piece to be sent back into the swarm, it must have been checked by at least one peer. If hash check fails, the piece is not sent back in. If hash check succeeds, the piece is sent back in. Your client does that, all other clients do that.

Since those peers have checked the hash on all the pieces they sent you, they must be good pieces. It follows that between those peers' connections and yours is a problem that renders these pieces bad. Possible sources of the problem is: Your machine, any node between you and those peers.

That many nodes between you and those peers go bad simultaneously is a very low probability. That many peers have problems that cause piece corruption simultaneously is a very low probability. That your machine has a problem is a much higher probability compared to the other two possibilities. What if the hash info you have stored for that torrent has gone bad for some reason? What could be the reason this hash info has gone bad? I can name a few, system memory, motherboard components gone bad that cause errors not detected or corrected but not properly, connection line noise not properly corrected, etc. All of these problems has happened to me in the past and I know what it does to data.

PM me the link to that torrent, I'll have a better idea of what's going on with it. I have never had a problem with hash check on my side of the equation and my machine is in perfect health so I'm certain that if there is a problem with that torrent, its source will not be me.

ML

p.s. Lay off the personal attacks.

OK,

That was a more coherent reply compared to your original one.

What you should remember though is that what's sent each time is a 16 kB 'piece of a piece'. In a 4 MB piece torrent there are 256 of these 16 kB pieces any one of which could have come from a number of different peers to make up the complete piece.

Sure, for the piece of the piece to be re sent, the piece it belonged to would have needed to have been hash checked first.

If the router mangles the sent data packet having assembled a CRC correct and in all other respects 'valid' TCP packet, thereafter any other node in the network will of course accept it as valid!

Perhaps there is the cause?

Why though are 13/14 of the corrupt data senders uTorrent clients, when there is an almost 1/3rd equal share of Bit Comet, Azureus, and uTorrent clients?

If I ban the corrupt data senders, why does the torrent become non-corrupt again?

How can you possibly blame that on my set-up?

Any way. I'll try and PM you the torrent URL. Please be patient however. 4 MB pieces take a long time to complete! I'll also send you the list of corrupting peers and you can compare.

[Martin Levac]

Here are partial results for this torrent so you can compare with what you got and perhaps find the solution.

My settings:

version 1.4.1b405

Max conns: 65

upload slots: 10

Max upload: 65kB/s

Time elapsed: 1h 35m

DL'ed: 138MB

UL'ed: 219MB

SR: 1.5'ish

Wasted data: 6MB, includes 1 hash fail + some cancels. Hash fail happened after first piece, thank you.

Time to get first piece (4MB): 40 minutes, ridiculous. Imagine if that piece was bad?

As I write this, I just got another hash fail for a total of at least 10MB out of 154MB, that's a pretty bad S/N ratio. Taking into account Lexx's experience with the same torrent, I'd have to waste at least 14x20MB=280MB to get rid of all the bad peers (if there are not more) if I only use uTorrent's bad peer/temp ban mechanism. I could reduce that number to 0 if I use PG and manually ban those IPs that Lexx sent me.

Nothing bad on my machine and I appear to get similar results so I have to believe you Lexx that there's nothing bad on your machine either.

Judgement call. This torrent is bad, period. Its pieces are too big, its files are too big (first-and-last priority kinda thing), start up is too slow, it's S/R hell all the way since I don't see any speed increase, whatever I set my upload to. It appears that the majority of peers have no clue how to configure their clients for good performance. Multiple bad peers send corrupt pieces. All this put together make this torrent effectively useless. On top of that, the torrent is DHT'ed and multi-tracked even if it comes from a registration site.

Eh, third hash fail just now and I'm not even at the 200MB mark. POS torrent, just get it somewhere else, there are many sites with absolutely better peers than this one. It's all in the email I'm about to send you, Lexx.

Finally, I think it's not uTorrent's failure, in fact I think uTorrent works as intended by properly hash checking the bad pieces you and I are getting.

Pfft, fourth one, exact same piece as third, at the 210Mb mark. Waste at this time 2h since start: at least 16MB.

ML

[Lexxington]

@Martin Levac,

Hi again and thanks very much for doing the test.

Did you also take note of the client that the hash failed peers were using?

Yes, I agree with you about how most clients don't seem to have their set-ups tuned properly for good performance.

This torrent (and the series 1 and 3 torrents also) as I said originally seem to work work better with BitComet.

This seems to be because as you said it's DHT'd and BitComet finds and utilises much more peers to work with and more importantly more seeds. Maybe the Peer exchange does the trick?

On BitComet with this torrent, I get around 14 connected seeds and over 200 leechers. With uTorrent, it's around 3 connected seeds and 50-60 connected leechers.

I can get an individual file selectively downloaded in about 5 hours or so on BitComet with a speed varying from about 6-45 kB/s. On uTorrent I rarely saw the speed rise above 8 kB/s.

On Sunday, using uTorrent, I wasted 2 hours with no data at all downloaded because of hash fails.

Anyway, I disagree that the torrent is bad per se. Yes, the 4 MB piece size is ridiculous, more especially when lots of peers are sending corrupt data!

Also, just WHAT is the reason for the corruption?

Do you think the corruption is deliberate or something else?

I started a different thread in the 'Feature Requests' section asking for an optional selection of number of hash fails before a ban to be considered, but some moderator decided to close the topic for whatever reason known only unto HIS self.

If uTorrent was a lot stricter in this area, the amount of wasted data could be cut down considerably.

Whatever the reason for the bad data being sent, allowing it to propagate is bad for the torrent for everyone involved.

I see that NiteShdw posts here now. Those torrents were from his site and I put in a request for someone to re-seed a NEW set of torrents with a 512kB or 1MB piece size to replace the 4MB piece size ones but there were no takers.

Anyway, it would be interesting to get to the root of the corruption especially if the corruption is not DELIBERATE but unknowingly done.

Regards

Lexxington.

[Switeck]

Also, just WHAT is the reason for the corruption?

Do you think the corruption is deliberate or something else?

The corruption is almost certainly deliberate. There are whole corporations dedicated to that task. Media Sentry being one of the most active right now, although I'm sure BayTSP is probably behind it as well. I consider their activities to be an encroachment on freedom of the press, the computer. I did research for Peer Guardian's blocklists awhile back, and continue to recheck parts of blocklists I download from time to time.

The fact that torrent chunks are 4 MB in size means just 1 byte of corruption can force a 4 MB redownload, so it works in the corrupter's favor.

With the latest Peer Guardian 2 blocklist, (possibly converted for use as µTorrent's ipfilter.dat to prevent needing to run Peer Guardian 2) you'd probably finish the torrent pretty quickly -- though even with it some corrupters would slip through the block.

With a dedicated effort, you could even figure out who some of the corrupter ips are and manually ban them (with ipfilter.dat in µTorrent...or some other means) even without getting a blocklist made by someone else.

[Lexxington]

Also' date=' just WHAT is the reason for the corruption?

Do you think the corruption is deliberate or something else?[/quote']

The corruption is almost certainly deliberate. There are whole corporations dedicated to that task. Media Sentry being one of the most active right now, although I'm sure BayTSP is probably behind it as well. I consider their activities to be an encroachment on freedom of the press, the computer. I did research for Peer Guardian's blocklists awhile back, and continue to recheck parts of blocklists I download from time to time.

The fact that torrent chunks are 4 MB in size means just 1 byte of corruption can force a 4 MB redownload, so it works in the corrupter's favor.

With the latest Peer Guardian 2 blocklist, (possibly converted for use as µTorrent's ipfilter.dat to prevent needing to run Peer Guardian 2) you'd probably finish the torrent pretty quickly -- though even with it some corrupters would slip through the block.

With a dedicated effort, you could even figure out who some of the corrupter ips are and manually ban them (with ipfilter.dat in µTorrent...or some other means) even without getting a blocklist made by someone else.

Hi,

It's a long thread so you probably missed it, but I've already said that I use PG2 and it is auto-updated every day with the latest p2p, ads and spyware block lists. The corrupting peers here aren't on those lists.

Yes, adding the IPs of the corruptors to a separate PG2 block list does indeed enable the torrent to procede without any further hassle.

Unfortunately, this doesn't help anyone else in the swarm.

So, the crux of the matter is not so much how to cure the problem but WHY the problem is occurring in the first place (assuming that the corruption is not deliberate).

Alternatively could uTorrent be better designed to cope with this sort of corruption deliberate or not?

[chaosblade]

Lexxington, I suggest you take care in future postings. All your posts in these forums so far have a very disrespectful attitude towords people and the client itself in the sense that you refuse to believe that anything is ever wrong in your side, or MIGHT be.

[Firon]

1 byte of corruption won't work, it has to be at least a 16KB block

[Lexxington]

1 byte of corruption won't work, it has to be at least a 16KB block

Actually 1 bit of corruption would cause the piece to fail!

The blocks are 16kB and are not themselves hash checked, so any part of any block that gets corrupted (however small that corruption is) causes the whole piece to fail the SHA-1 hash check.

@chaosblade

I suggest you re-examine your own attitude.

The client is a piece of software. 0's and 1's. If you want to pay respect to inanimate objects then I would see a shrink if I was you.

If you can't be objective and want to treat software as your 'pet' then I suggest you get a Tamagotchi.