High priority pieces stuck on ultra-slow dl.

[ShawnFessenden]

I guess this is more of a phenomenon than a problem, or maybe it's a potential tweak. This is a longstanding issue across several releases of uTorrent & 2 "versions" of Windows (2k, XP).

The Scenario:

A torrent that consists of several files. In my case, I'm talking about a series of video or audio: a lecture series, a play on CD, a season of a particular sci-fi or horror show - whatever. Several files that are to be watched or listened to in order.

I always set priority on the first file high, the second normal & the third low, then set the others to skip in order to concentrate activity on the first three. When the first is done, the priority settings move down one: second => high, third => normal, forth => low.

The Issue:

The problem is that the last few pieces of the high priority file will sit there for a very long time. When I switch to Pieces view to see what's going on, the lowest piece numbers (the last pieces in the high priority file) are stuck on the lowest speed connection possible. As other pieces download with no trouble & fly past, completing just fine, these pieces just plod along taking forever to complete.

The Solution?

I don't know enough about the protocol (or the application) to be able to comment further, but it seems to me it should be easy to recognize this condition & switch the piece to a peer that is performing better. IMO High Priority *should* mean high priority on everything - not just socket or thread pool concentration.

Further Suggestions:

In fact, for filling in trouble spots I don't see why it's necessary to cling to the BT protocol at all. Look at the best overall performer, open the file directly via a pseudo RPC wrapper & just SetFilePointer & ReadFile. Have the completion port do whatever kind of error checking you want & dispense with the whole BT idea - for small fill-ins only obviously. I can't count the number of times I've sat there waiting for the last 4M of a file to come in on a high priority file while normal & low priority pieces just fly by.

-SHAWN-

[Switeck]

So your solution is to sabotage the already weak tit-for-tat (in the BitTorrent protocol) further?!

Priority on your end alone has little effect on who is willing and able to upload to you.

But if you want to test an external "workaround", disconnect the slow peers uploading to you using TCP view (or similar program) or ipfilter.dat in uTorrent. Good luck figuring out which one/s are to blame.

[ShawnFessenden]

> So your solution is to sabotage the already weak tit-for-tat (in the BitTorrent protocol) further?!

I know nothing about that at all.

> Priority on your end alone has little effect on who is willing and able to upload to you.

What has that got to do with anything? View the currently on-line swarm members as a potential source pool and the logic is ULTRA SIMPLE:

IF a high priority piece is on a connection that's classified 'slow', AND that piece is available from a different peer that's preforming better, THEN switch to that alternate peer. What's the problem?

-SHAWN-

[paintball9]

uTorrent requests rare pieces first, to avoid having the torrent die mid download. However if a 'fast' peer is only uploading low priority pieces its possible that they don't even have the high priority ones. They're just uploading what they have. Trust us when we say that what you want to do is a bad idea. Don't fix it if it ain't broke.

Also priority doesn't mean that that's the only thing utorrent will strive to get. From my experience, high priority files make up about 75 percent of piece transfers, normals is around 15-20 and low is somewhere around 5, (if the torrent is all seeds). Basically it's always gonna have some pieces transferring that you don't necessarily need first. It helps the protocol out.

[ShawnFessenden]

uTorrent requests rare pieces first, to avoid having the torrent die mid download. However if a 'fast' peer is only uploading low priority pieces its possible that they don't even have the high priority ones. They're just uploading what they have. Trust us when we say that what you want to do is a bad idea. Don't fix it if it ain't broke.

Ok I see. So assuming random (homogeneous) distribution of pieces across a peer swarm, this is definitely a bad idea.

So anyway - never mind. Until I'm completely familiar with the protocol I guess it doesn't make sense to draw conclusions from observations. So - put it down as a negative observation, no matter what the cause: the effect is way undesirable, regardless of why it happens.

-SHAWN-

[Switeck]

Yes, there's a problem...it grows exponentially worse as more people do selective downloading.

Test an external "workaround", disconnect the slow peers uploading to you using TCP view (or similar program) or ipfilter.dat in uTorrent.

[ShawnFessenden]

Yes, there's a problem...it grows exponentially worse as more people do selective downloading.

Test an external "workaround", disconnect the slow peers uploading to you using TCP view (or similar program) or ipfilter.dat in uTorrent.

Well I'll be testing off-line in an isolated "vm swarm" of my own. I'll be messin with things pretty heavily using my own client & so don't want to piss people off. I mean I'm bound to screw some stuff up early on!

Yes - I'm getting there. I have most of the protocol theory under belt now. In particular, I understand your allusion to 'tit-for-tat' (hehe!).

Just going through algorithmic details now & testing tracker ops.

Tomorrow I'll move on to testing protocol ops. These will probably take a while to get in order. I'm hoping to have good local full-client tests running in a couple weeks, at which point I'll be able to start casting around for a good solution to the serial scenario (I've officially dubbed it 'cereal mode') without bothering anybody :-)

If I actually find anything decent I'll give it a try in the wild & see what happens on my isp's local node between me & a few friends. If it looks interesting & doesn't involve too much overhead I'll pass on the details.

-SHAWN-

[ShawnFessenden]

Well well well. Whaduyano. 2.2: Prioritize by File Order. Apparently somebody didn't think it was such a dumb ass idea after all huh?

-SHAWN-

[DreadWingKnight]

Doesn't make it any less of a bad idea.

[wowed]

Actually, restarting the torrent will generally fix this issue. I have noticed that upon restarting the torrent, the client requests the Pieces currently marked as the highest priority first to the fastest supplier available. So by inference:

1 Yes this is an issue.

2 This has NOTHING to do with piece homogeneity or anything of the sort.

3 FIXING THIS WILL NOT KILL THE SWARM.

These points are self evident because if the piece was indeed a problem piece, then upon restart, the piece would be relegated to the slowest connection while the rarest piece on the next highest priority file would be serviced by the fastest peer with the piece.

I will paraphrase Shawn's intent here with much liberty. I think that Shawn is not talking about prioritizing specific pieces OUTSIDE of the currently allowed prioritizing spec which is generally agreed to be a bad idea. I think he is asking that the final pieces of a file that has HIGH priority be linked to either medium or fast distributors not the slowest ones in the swarm.

Unless I misunderstand the protocol completely, the point of BT is to allow clients to download at the highest speed possible from the swarm while maintaining a homogeneous mix of available pieces. So, if homogeneity has been achieved by the time he started downloading, then as many of the pieces on high as possible should be downloading with the fastest seeds or peers available to him, and when it is down to the last piece THAT should download with the fastest peer, no excuses. If homogeneity had not been achieved at the time he started downloading, then he should be seeing preference given to his highest priority files while the pieces therein are downloaded according to rarity. HOWEVER, after every piece in the file IS downloaded, then the LAST PIECE of the file should be switched to the fastest peer available.

This solves two problems at the same time. First, the ability to prioritize files is useless if the files end up having pieces missing with an amazing and pretty Gaussian distribution. Secondly, it should balance the load of uploading pieces from slow clients to clients that have a download speed commensurate to theirs. In other words, fast uploaders can be linked to a piece for slow downloaders. However, slow uploaders cannot be linked permanently to a piece for fast downloaders when faster peers are available unless there is good reason for it(rarity or faster uploaders are occupied). As this blocks faster peers from uploading the piece. Doing so would slow down the swarm speed and will slow the acceleration of the availability ratio.

This problem exists in absolutely no other client I know of, sooo.. I am guessing it is unique to Utorrent.

[ShawnFessenden]

> I will paraphrase Shawn's intent here with much liberty.

Ok cool. I think that what's at issue is the philosophy of rarest first itself, in this specific instance (serially ordered files). It's not appropriate IF the files are to be filled out from first to last. Certainly, within a specific "window" it is - the vision is to create a weighted "wave" of probability that travels from first piece to last, the frequency depending on say three or four times the average pieces per file (starting with the assumption that most of the files are of similar size). Within this window, rarest is applicable.

Naturally the size of the swarm & availability AFTER considering chokes comes in to play, but exactly how is the hard part. My own tests with a 16 machine network (with a 1:3 seeds to peers ratio) & programmable strategy engine hint that this idea MAY be worth pursuing, but it's not ready for prime time tests yet & may actually come to nothing in the real world.

The issue of handling end game (on a per file basis) is a separate consideration. Currently, I'm giving pieces to complete the "next" file highest priority in that they are requested from the fastest servers first, and one third of the download threads are dedicated to them. This is obviously where the unacceptable wasted bandwidth factor comes from: as soon as a file totally completes all pieces for that file on other servers are deleted from those server's pipelines, or are canceled if in progress or are just wasted if cancellation is not possible.

There's an architecture fork here. Right now it's one thread = one socket, but it's probably a better idea to send ready sockets to ready threads, sort of like a Win32 completion port but different. I'm not at that level yet. This would require a higher priority management thread that I'm not really willing to get into, and W32 specifics are out of the question.

So far, in an admittedly limited & sterile local network, it's working out to better than six of one and a half dozen of the other. If these stats hold up in the real world THEN I'll start making noise. One consideration is that I have yet to do obfuscation, which will (probably) be necessary if I'm to actually do tests on my ISP. These tests will be carried out on PD torrents but I'm pretty sure they'd kill em if they noticed.

-SHAWN-