Just a comment on queueing and hash fails, again

[z9999]

Watching a 3 gB torrent that has been running for 16 days now, averaging about 3.7 kBps having 1 seed and 9 peers nearly constantly this is what I have seen. And the seeds and peers have changed over this time but seem to remain at about the same count each time I look.

I currently have 37 pieces in queue at various stages of completion, from 0-254 blocks completed. I have 8 pieces with over 200 blocks completed and they have been that way for over 12 hours now. It appears that as pieces reach 253 or 254 blocks completed a new piece is queued to take their place upon completion, but instead of completing they remain incompleted as their availability seems to take precedence over their being completed and a piece with lower availability begins to be requested instead. This seems to be the norm when piece sizes are 4 mB and as such hash fails takes an enormous toll on these torrents. This particular torrent being 3 gB has accumulated 322 hash fails, (1.288gB), and is only 83% complete. In my case I seem to encounter hash fails periodically related to transmission path problems, and if I see such a problem to begin I usually stop all running torrents and wait and try again later. Torrents with 4mB piece sizes suffer greatly as often nearly every piece in progress has a corrupted block by the time I can identify a problem exists. Queueing an enormous number of pieces in relation to the availability of seeds and peers, as well as average DL speed increases the data losses that occur. Having 37 pieces queued amounts to requesting 144 mB of data to be sent and if the average DL speed is running 3.7 kBps and is equivalent to about 10.8 hours to accomplish if no new pieces are added to the queue.

Obviously I'm in a minority as far as this being a problem, but in my opinion it makes very little sense to request a new piece be queued when there are already more pieces queued than there are peers/seeds to complete them, and most of all if a piece needs but one or two more blocks to complete, why not just complete it and then look at the availability of the other pieces. Leaving a number of pieces 16-32 kB shy of completion for 12 or more hours for the reason that another piece has lower availability is just setting up a possibility of numerous failures. In many cases it appears that the only reason a piece gets completed is because enough peers who had the same piece drop off causing it's availibility to be among the lowest, and at that time often it fails hash check allowing me to know that I have had a bad connection some time during the previous 12 hours which may have contributed to all queued pieces eventually failing hash check.

Curiously, I have noticed that torrents with piece sizes of less than 4 mB don't appear to queue up more pieces than one might reasonably expect to be queued based on the number of seeds/peers, and they always appear to have fewer pieces queued than the total number of seeds/peers.

If any others have experienced similar results I would like to see their comments as I expect this is something that will be ignored unless it can be proven that it affects a number of users.

[Switeck]

It sounds like the worse-is-better philosophy I've seen in other forms.

Lots of people have "discovered" they get slightly better download speeds if they allow lots of connections. By lots, I mean 100's. They discovered that setting their upload speeds too high slows their downloads, so they put their upload speeds as low as possible to still allow "fast" download speeds. When they can't max out their download from 1 torrent, they run a few more torrents at once to get more 'bang for the buck'.

This aggrevates the problem of hash fails because with lots of connections on a well-seeded torrent there is always a few per every X peers "willing" to give you data on demand. The problem is, their average speed to you is always dismally low. The reason is: they have a HUGE number of upload slots. They are willing to send out to more people because they have more upload slots than currently in use. But each new upload slot divides their limited upload bandwidth into smaller and smaller fractions. I'm not talking 1/2 KB/sec...these speeds can be even less than 100 bits/sec! But such slow transfers can mean the connection times out trying to send just one 16 KB piece.

As a rule-of-thumb, I'd say probably more of these connections are firewalled than not, simply because they are blissfully unaware of the problems they're causing or even that they're firewalled.

A simple way to avoid data from firewalled users is to be firewalled yourself. But even that's not a solution, because µTorrent is unreliable from what I've seen while firewalled.

In my tests, while firewalled on both larger and smaller torrents, my upload speed is completely erratic -- often running at only a fraction of max allowed speed. Connections drop so often that I'm convinced the BT protocol itself is dropping them due to "inactivity" of any kind. On small torrents, I have a very hard time connecting to peers -- and when I do, I disconnect from them and take a couple minutes to reconnect. On large (lots of peers+seeds) torrents, I get sufficient peers connected -- but they all seem overloaded and not interested in getting pieces of the torrent from me...or do so at a very bursty and slow rate.

µTorrent's advanced settings might reduce some of these problems...but will probably create a couple more in the process. Such as net.low_cpu=TRUE supposedly makes fewer requests at once. Lowering net.max_halfopen and bt.connect_speed might be an indirect help. Definitely set bt.prio_first_last_piece=FALSE, as trying to get those WHILE trying to get everything else means making more rather than fewer requests.

[z9999]

My settings have been:

Global max 200

Max peers 15

UL slots 2

I just recently changed them to 75, 5, and 2 respectively.

So I don't believe my settings were too high previously, but the lower settings have shown some improvement. I had already realized that increasing activity was not an answer to the problem as it is apparent that with more peers connected a larger queue was created initially and as pieces neared completion they would be put on hold rather than completing and in addition if within 1-2 blocks of completion it appeared that a new piece would be added to the queue as well.

You might check my most recent post which I made just prior to reading this, commenting on the results of the changes I made about 16 hours ago and their effect.

I appreciate your response very much and the enlightenment on the advanced settings which I will take a look at as well and see if any of them may produce more positive results.

Another thought which developed during this process is that although it appears that I am banning peers frequently due to has fails, I don't appear to be banned as a result of involved in hash fails of pieces I contribute to by others. Although I may ban a peer for having received bad data it would be nice if I could continue to send to that peer if my data sent is arriving uncorrupted. Could it be possible to instead of banning a peer cease to request pieces from the peer instead? Currently the only solution I have found is to wait until the entire torrent is finished at which time I can send to all peers without them being banned.

Just checked the settings you mentioned, which I have never tampered with, and they are as follows:

net.max_halfopen = 8

bt.connect_speed = 20

bt.prio_first_last_piece = false

If you think changing any of them would be helpful please let me know which and what value to try as I am unaware of their details.

Thanks again.