Why doesn't seeders send the rarest parts?

[SirLucifer]

An example:

I have an ongoing download with only a few seeders (3) and a couple more leechers (19).

Whenever one of the seeders are sending me data, it's almost always one of the chunks where there are more sources.

Wouldn't it be a lot smarter and more efficient for the seeders to send the parts where there are few sources?

Maybe, in that case, the downloads will actually be done quicker as the seeders often are a lot slower than the leechers.

In my current example, the seeder is sending at 0.1 k/s and only once or twice per hour, while leechers send at 5-800 k/s, depending on origin and network.

To me, it looks like I will have to wait for the slow seeder(s) when all chunks with multiple sources are transferred.

[DreadWingKnight]

That's what initial-seed or superseed mode is for.

[SirLucifer]

Great! And what about the average user who just leave their clients running on the initial settings?

Why not implement some smart into the clients?

[DreadWingKnight]

Most smart clients already preference requesting rarer pieces. It's not possible to enforce a rarest-first method without using the initial/super seed mode.

[Ultima]

And "initial seeding" != "initial settings"

There is no way to force peers to do anything. If a peer doesn't request a piece, the seed can't send it. BitTorrent's Fast Peer extensions have a method of suggesting pieces to peers, but peers don't have to accept the suggestion. Initial seeding is similar, only, the initial seeding client pretends only to have a single piece so that peers have no other choice in pieces than that one piece. So telling us to make the client "smarter" (than has already been implemented) with regards to seeding without further details is just you being ideological.

Telling us to make the client "smarter" with regards to downloading, without additional details, is again just you being ideological. µTorrent already prioritizes rare pieces, but there's no reason seeds have to send pieces to you in the first place either.

[SirLucifer]

That sounds really good, in theory.

But in this practical world it doesn't really come true.

This torrent I'm talking about (the one I've been monitoring a bit) has four online seeds of which two are µTorrents and they keep sending (when they are sending, which is very rarely) pieces that have at least six sources. Simple math six sources with four seeds leaves two other sources, which are leechers.

The ideal, in my mind, would be asking for the pieces with only four (in this case) sources, i.e. the seeders.

I'm not sure what more information You would need, it just seems like something has been overlooked in the negociation of parts. Or, as You said, seeders doesn't *have* to follow requests, but why not? The requesting client knows which pieces are low-seeded, right? Any request on those pieces should be followed to, in the long run, speed up distribution of contents.

[jewelisheaven]

THEY may not be using initial seeder mode.

[Ultima]

Indeed, theory can't always (and often doesn't) predict real-world cases. I don't really have any other explanation to offer as to why seeds are disregarding rare piece requests. And note that I'm not saying that this can't be improved (I don't know for certain myself), but just that it's easy to say "improve it" without explaining exactly how one would go about actually doing that.

Edit: Also, while µTorrent does prioritize requesting of rare pieces, that doesn't mean it exclusively requests rare pieces.

[SirLucifer]

I am truly aware of that dilemma, people often say "fix this problem" but it's never that easy. I know, since I'm a programmer myself. But, I thought that if I put this problem out there it might root itself in Your subconcious and maybe a solution will reveal itself in due time. And since I'm not really keen on digging into the protocol specifics, I just, kind of, hand it over to You guys...

[Ultima]

Fair enough

I can't offer any real explanation because I myself am at least as confused as you are, and don't really see why a seed would actively choose to ignore certain piece requests, but not others.

[hermanm]

> Wouldn't it be a lot smarter and more efficient for the seeders

Makes you wonder if sending pieces in cache is more efficient than pieces that are very rare (e.g. need to be read from disk). If only rare pieces were sent, disk contention could potentially exist if the hardware can't retrieve requests fast enough.

Bittorrent so needs universal multicast support!

[DreadWingKnight]

Bittorrent so needs universal multicast support!

ISPs didn't want to do multicast.

They got BitTorrent instead.

[Ultima]

@hermanm: Yeah, I thought of that as a possibility as well, but I don't see disk IO as a significant bottleneck in most cases, unless the seeder has ginormous amounts of upload bandwidth to push data out -- faster than the disk would be able to keep up with (rare). Disk IO shouldn't be an artificial bottleneck for "good" piece distribution any more than it should be an automatic, artificial bottleneck for download rates (automatically self-imposed by µTorrent, but not due to disk overloads, that is -- which, thankfully, µTorrent doesn't do).

BitTorrent already can work similarly to multicast, especially with initial seeding. One peer requests and receives the data from the seed, and can then share it with other peers that request the data -- seeder shares once, many people can receive (basis behind initial seeding). The hypothetical peer that first requests the piece would then be acting as the router. Forcing BitTorrent to work strictly using multicast techniques would only be another artificial limitation that likely brings with it more complexity and/or problems than it solves (whatever it is you're trying to solve with it...?). And uh, I'm not really seeing how multicast is relevant to SirLucifer's problem, as it wouldn't affect piece rarity.

[BlueDragon]

Old thread but still very present topic...

I have been observing several weeks a 28GB torrent which I have been reseeding again as peer not having the full content of a set of 4 DVD's (e.g. selective download).

Out of it (amongst other experiences) I can confirm for sure exactly what SirLucifer stated above: uT running as peer in a torrent swarm with many other uT clients amongst them (so compatibility should not play a part) does not prioritize according to the rarest peaces and is has NOTHING to to with inital seeding at all!! Would it have been linked with inital seeding a sequential piece exchange would have been clearly visible which was not the case at all according to my watching over weeks (incl. logging the traffic).

[Ultima]

does not prioritize according to the rarest peaces

And how have you determined this? By logging the request packets between µTorrent and other peers? As well, have you taken into account the fact that bt.prio_first_last_piece can skew results?

[BlueDragon]

Yes as I said I watched the traffic exchange by using the (basical) traffic logging function of uT. Unfortunately it does not tell me what is the aggregated data which has amongst others been exchanged but I could clearly see the piece/block requests, deliveries, have, chokes, rejects etc.. enough at least to be sure of my statement.

bt.prio_first_last_piece could of course skew results as well as setting the priority of certain files to low or high instead of normal. In this case I did not change the defaults settings though. If you want to get evidence of it I can send you the magnet URI link. (It's a reseeded swarm of partly selectively downloaded stuff with almost no seeds left most of the time so it's a particular case...)

[Ultima]

Yes as I said I watched the traffic exchange by using the (basical) traffic logging function of uT.

µTorrent log != packet log

The Logger tab does not show anything about what pieces µTorrent requests from other peers. So again, how do you know µTorrent didn't request the pieces, and it just turned out that the peer didn't bother sending it? I'll reiterate:

There is no way to force peers to do anything.

Just because you don't see a piece being downloaded doesn't mean µTorrent didn't ask for it. And just because there is a rarer piece out there doesn't mean µTorrent should ignore common pieces too, so to hold that against µTorrent is rather silly.

[BlueDragon]

The Logger tab does not show anything about what pieces µTorrent requests from other peers.

Well... Let me quote some example out of the logger:

[2009-11-27 03:10:47] XX.XXX.103.244:62969: [Torrent 1.8.2 (88.4)]: Requesting 5839:1392640->16384

[2009-11-27 03:10:51] XX.XXX.103.244:62969: [Torrent 1.8.2 (88.4)]: Got Piece: 5839:1359872->16384

[2009-11-27 03:10:51] XX.XXX.103.244:62969: [Torrent 1.8.2 (88.4)]: Requesting 5839:1409024->16384

[2009-11-27 03:11:02] XX.XXX.103.244:62969: [Torrent 1.8.2 (88.4)]: Got Piece: 5839:1376256->16384

[2009-11-27 03:11:02] XX.XXX.103.244:62969: [Torrent 1.8.2 (88.4)]: Got Choke

[2009-11-27 03:11:02] XX.XXX.103.244:62969: [Torrent 1.8.2 (88.4)]: Got Reject: 5839:1409024->16384

[2009-11-27 03:11:02] XX.XXX.103.244:62969: [Torrent 1.8.2 (88.4)]: Got Reject: 5839:1392640->16384

If this does not to allow me to know that my client requested to client XX.XXX.103.244 running uT v.1.8.2 having 88.4% completion at a given date and time the piece # 5839 (actually one of it's 16KB blocks) than you must explain me what it really does...

Who is talking about forcing any peer here? My statement was that obviously my client is requesting a piece out of all the pieces (according to the handshake and subsequent have messages which my client got from the other peer) that the peer in question can offer. I conclude (maybe wrongly but then correct me and give me good arguments as proof) that my client is requesting some piece first and the peer has the choice of sending it or not (choke me and reject my request as at the end of the example above). Did I say my client would force the peer to do something?

Thank you for the "rather silly" and ask yourself what is silly here (without becoming to emotional). As a matter of fact the exchanges in this forum are far to often being led by the Ego of some of the admins / mods than something else. It's a pitty. Facts are facts and you are welcome to communicate them without flaming or moving the thread to the trash. We should be a community to exchange and learn trough it. There's no silly question or even "rather silly" idea but only some people too much thinking that they are the Gods of the knowledge. That's at least my clear feeling having been reading in this forum for many months now. There's room for improvement...

Thank you and think about it if you want, this cannot be forced upon you. :cool:

[Ultima]

You're right, that excerpt from your logs do show the request; apologies for claiming otherwise, as I haven't ever observed it in my logs. Admittedly, I don't watch the logs enough to be able to claim absolute, domain knowledge on and familiarity with its output, but I watched µTorrent download the entire OpenOffice.org torrent without ever having seen any such message -- all logger options enabled.

And so, that begs the question: can you be sure µTorrent is logging each and every request? If I haven't ever seen messages of the form you pasted in my logger, despite the fact that I'm downloading pieces (and as such, clearly requesting pieces), then who's to say the fact that you haven't seen any requests for rare pieces indicates that there hasn't been any such request?

If you really want to get down to the bottom of this, then WireShark packet exchanges and determine whether µTorrent really isn't requesting rare pieces that way, because µTorrent obviously doesn't log everything in all cases.

On forcing, I never once said you said anything about forcing. I said that just because µTorrent isn't downloading a rare piece doesn't mean it hasn't requested it because there is no way to force a peer to send you a rare piece. Why would I talk about downloading when you were talking about logs? Because I (incorrectly) assumed the logger didn't log requests, so the only remaining tangible method of determining whether µTorrent is requesting rare pieces would be the Pieces tab, but of course, that is useless if you can't force downloads -- hence the quote.

As for the bit about "silliness," I don't see why you're taking such offense to it. It wasn't even an attack at you. Thanks, but I'm going to pass on responding to the ad hominem; I'm not interested in strawman debates.

[BlueDragon]

You're right, that excerpt from your logs do show the request; apologies for claiming otherwise, as I haven't ever observed it in my logs.

I welcome your apoligizes but ad hominem or not, strawman debate or not (from you point of view) I think you should know what you are talking about even more because you are an admin of this forum and as such you bear a greater degree of responsibility than a member. This being said let's go back the the technical facts.

As nobody could ever explain me nor could I found any extensive help about the usage of the uT inbuild logger I can understand that you did not see any such message... Personal experience and hours of fiddling around are making the difference in this case I guess. The key to the sesame is the right click on the concerned peer and the selection of 'Log traffic to logger tab'. The flags seem not to be linked with it, very unlogical (from my point of view of course) but apparently true...

can you be sure µTorrent is logging each and every request?

If I cannot be sure what's the use of it after all? Out of what I could see in my traffic logs it appeared very clearly that the requests or rejections of the requests are logged. So building on this it has logically to be expected that this is true for all the same type of information. Would it not be like that I would simply call it a joke and the logger would be utmost unreliable / useless. Maybe there are other types of information exchanges which cannot be logged but this is not changing anything concerning the priorization of rare pieces as initially discussed in this thread.