Jump to content

rqst..the ability to download the first half of a file first


HRCX

Recommended Posts

Hi, I don't know if this is even possible given my limited understanding of the nitty gritty details of the bt protocol -- however, if it is I would be extremely interested in seeing it... (and would also be willing to donate some agreed upon $ for it before & after it'd be implemented).

Anyway, what I'm wanting / hoping-for is this:

I want to be able to have a client that will allow me to tell it to (on a particular file that I've started downloading, not on all files) download the full first half of the file first, before it ever starts working on the second half. I realize that there could be a semi-unlikely instance of there being only some part of the second half currently available to get, in which case it obviously makes sense to go ahead and download that / those part(s), but in such case I would want the client to still be looking more keenly / urgently for (and thus trying to download the most) those parts which belong to the first half of the file.

Now to why I want that feature..

I download avi files the most (with bittorrent anyway).

And there are a number of programs / various ways in existence which have the capability of allowing one to play a partially downloaded avi file.

However, for that to work, most of them need the beginning of the file to be complete (the avi header, like the first few dozen kb of it or something, I dunno) - and so anyway, even the ones that don't need the very beginning of the file to be perfect still need the file's data to be mostly intact w/ out a bunch of random / pseudo-random gaps in it, like will be currently found in a partially (half) downloaded bittorrent download (using any currently existing client, to the best of my knowledge).

Whereas, if a client were able to allow the user to request it to implement the downloading of some particular file being downloaded in such a way as to download the entire complete first half of the file being downloaded first, it would then be possible (assuming the file in question is an avi file) to play the first half of it with little difficulty and without having to wait for the second / last half of the file to be downloaded.

Lastly, if such a feature would be possible to implement, it would theoretically be good, or at least fully neutral, rather than at all bad for the BT network / for the particular swarm being downloaded from, right? Since, if a user downloaded the first half, they could and would (of course) then continue normally downloading the second / last half of the file by the chaos-piece-availability scheme until it was complete, while they would have the entire first half available to share and get people who started dl'ing after them further along w/ the first half faster and probably further along faster period by having a lot of sequential pieces to offer. It would also probably only be a good or neutral thing, because most downloaders (in a particular swarm) probably wouldn't end up using it, so it probably couldn't really have much positive or detrimental impact of any kind on the swarm overall. (Although I think many users of utorrent may use it if it were added.)

Thanks for reading! :)

Link to comment
Share on other sites

This would be neutral at best, more than likely negative for the swarm. The seeders/peers try to give out rarer pieces first to make more unique copies faster. Downloading the first half of the file, which would almost certainly violate the BT spec, since it wasn't designed for sequential downloading (correct me if I'm wrong), would be contrary to that goal.

What do you need HALF the file for anyway? A small amount is all you'd need to see the quality, you can probably find a summary/review of the content elswhere, and most movie/tv releases already have separate samples for you to download. :rolleyes:

Link to comment
Share on other sites

If possible, i would like to rephrase the topic a little to avoid posting a new one: would it be possible to implement sequential piece downloading? Per file or per torrent basis is a choice for the authors :)

Besides:

This would end up working as sort of the opposite of Superseeding. Superleeching, I propose it should be called.
Care to elaborate on this? Sequential piece downloading is not (anti)similiar to Super Seeding in any way, this feature is indeed rather neutral.

Regarding the impact on the swarm (taken "per file" situation) - it has more or less the same impact as individual file prioritisation as much as i can see, and that is an ok thing considered by many...

Link to comment
Share on other sites

Downloading the pieces in sequential order would spoil the function of BitTorrent IMO, since it's more or less based on random numbers.

As I've got it, the seeder sends out a piece randomly chosen by the peer, who can send out that piece to another peer and so on.

However, if all peers would download the pieces in sequential order, all peers would get the same pieces. And this is in fact the same thing as superseeding, only the other way around.

Please, correct me if I'm wrong.

Link to comment
Share on other sites

If possible, i would like to rephrase the topic a little to avoid posting a new one: would it be possible to implement sequential piece downloading? Per file or per torrent basis is a choice for the authors :)

Besides:

This would end up working as sort of the opposite of Superseeding. Superleeching, I propose it should be called.
Care to elaborate on this? Sequential piece downloading is not (anti)similiar to Super Seeding in any way, this feature is indeed rather neutral.

Regarding the impact on the swarm (taken "per file" situation) - it has more or less the same impact as individual file prioritisation as much as i can see, and that is an ok thing considered by many...

Super-seeding works by sending out data chunks in sequence so that they get evenly distributed instead of giving the same chunk repeatedly to multiple peers. Your method would allow the same process to occur in the opposite direction, with the opposite results.

The reason it isn't "neutral" is that in all probability all the peers on a torrent would request the same chunks first, and then everyone would have the same chunks, thereby eliminating their ability to share missing chunks with one another and destroying that which makes the BT protocol unique.

Superleeching.

Link to comment
Share on other sites

Downloading the pieces in sequential order would spoil the function of BitTorrent IMO, since it's more or less based on random numbers.
The point of bittorrent is to avoid downloading one... thing from one place only. Sequential downloading doesn't prevent that.

Furthermore, all the posters assume worst-case scenario - all the users for some reason want sequential downloading. Let's assume the situation, where every person has prioritised the same file for example (because it seems to me that they are similiar in end result. again - if it's sequential per file, not per torrent. feel free to correct if i'm wrong). Wouldn't this create an abundance of pieces the amount of which covers a certain file? Cannot this be considered an abuse too?

Finally - yes, the sequential order is not as efficient as rarest first, but it is not leeching-efficient either..

Okay, i'd like to change the request a little - would it be possible to choose *only one* file at *one time* to download sequentially (strike out "per torrent")? This situation will be exactly the same as with prioritisation, only different piece ordering ;)

P.S.: super seeding is sending one and only one piece to each connected peer until it sees that piece on another peer. Given each peer enters a new swarm with sequential downloading order, the situation will hardly be different than in "rarest first" ordering.

Link to comment
Share on other sites

Okay, I didn't understand everything you said, Gribble, but I think you may still be missing out on the fundamentally unique feature of Bit Torrent, which is that peers can start uploading almost as soon as they start downloading. This can only work if different peers download the chunks of data in different sequences. And I am talking about the very beginning of a torrent here, when the only person with data is the original seeder. After availability goes up, things get a bit more flexible and leechers do less damage, but the principles still apply. If peers are allowed to download one file at a time (as they can already do manually in uTorrent, and automatically in certain other clients) the effect is that the torrent becomes divided into a separate swarm for each file. So, yes, the most popular files would become faster and the less popular files would become slower.

Whether this is a good thing or a bad thing, depends on your perspective. I submit that it is a bad thing, because it places more pressure on the original seeder, who should ideally be free to disconnect in the minimum time possible in order to deliver content elsewhere.

Edit: to look at it another way, availability would shoot up to .25 in much less time, but it would take exponentially longer to reach 1.0. If you've ever seeded anything from scratch, it's all about waiting for that 1.0 before you can relax.

Link to comment
Share on other sites

I have one request: could you please stop calling it "superleeching"? ;) It does not benefit a leecher in any way, 'cause a leecher is what? A person who takes everything without giving anything back. Sequential downloading doesn't help downloading faster and does give back, doesn't it? Superleeching just gives a very negative meaning to this feature..

Note: what do people think about my last proposal about only one sequential downloading file per torrent at a time? Would you go with that? It doesn't mess the swarm more than priorities and accomplishes the goal i seek.. There are variations possible too. For example download sequentially every second piece.. It would take me to the goal twice as slow, but i'm open to compromises ;)

Next,

I think you may still be missing out on the fundamentally unique feature of Bit Torrent, which is that peers can start uploading almost as soon as they start downloading
A little off-topic, but doesn't edonkey or winmx f.e. work in a similiar way? Mind you, i didn't look at the specs of them at all, so bash me to your hearts content if i'm wrong ;) It just sounds logical to me to do what bittorrent does and that's applicable elsewhere.

Furthermore,

And I am talking about the very beginning of a torrent here, when the only person with data is the original seeder
As much as i can see, the superseeding client can hand out pieces in whatever order it pleases (probably randomly initially. rarest first, if all pieces are present in a swarm). It can even send pieces sequentially at first ;) The point is for the seeder to upload as little duplicated pieces as possible, thus saving bandwidth. Clever.

Sequential ordering (for it to become dominant utorrent will have to gain majority in the bittorrent world and use "sequential..." for the whole torrent as default. not likely, and not my goal) will make pieces less frequent the further it is from the beginning, which will make a weird swarm, but not unusable, because all pieces will be present anyway.

And..

If you've ever seeded anything from scratch, it's all about waiting for that 1.0 before you can relax.
Sorry, i didn't ever seed from scratch, although i tend to revive torrents where no seeds are present anymore, but that is not the same..

I have a strange feeling that we will hear no response from the authors and will just duke this out among ourselves :P

Edit: fixed some spelling mistakes and removed some smilies. A little too much smiling ;)

Link to comment
Share on other sites

Well, I have to disagree...I also feel like most here that it looks a lot like Super-Leeching. The BitTorrent protocol is based on sending random pieces of a torrent...why would you want to mess with that? :?

Also, I think this is the wrong forum, I believe if you're really serious about this you should contact Bram Cohen personally and debate with him why you think your method is superior...'cause in here this is going no-where fast. Just my 0.02¢. :arrow:

Link to comment
Share on other sites

The problem with the philosophy that it works as long as not everyone uses it, is that it might get banned from trackers because of the inefficiency this would bring to the other peers.

And about calling it "Superleeching":

Superseeding isn't about a good speed. It's about low bandwidth as you already meantioned. That's why sequencial downloading would be very similar to supedseeding, only the other way around. ;)

Link to comment
Share on other sites

Also, I think this is the wrong forum, I believe if you're really serious about this you should contact Bram Cohen personally and debate with him why you think your method is superior
You are mixing something here.. The bittorrent protocol does not define a way you _have_ to download pieces, it just says that one way is more efficient than another if you decide to do so. And it's "rarest first", not "random" order that is more efficient and hence more widely implemented ;) And the method offered by me is not superior if applied for the whole torrent (again, i do not want this), i never said that :P

Next,

That's why sequencial downloading would be very similar to supedseeding, only the other way around.
Och! Be it your way. I'd really like this feature implemented whatever it ends up being called :D

Also, almost every feature can be abused liek this one, but isn't it about the "choice" in the end? ;) One could assume priorities is an evil feature too :)

None of you stated your opinion about that "one sequential file at a time" thing either or offerend any compromises... :(

Could i request a custom build then? :D (highly unlikely the authors would take the time to do so, but it doesn't hurt to try :) )

Link to comment
Share on other sites

Actually, defining how you must download a file is pretty much exactly what a protocol does. The word "protocol" litereally means a set of rules. You may be able to make a client that works without following those rules, but you will be violating the protocol and you will get your client banned from trackers. Now, I'm personally pretty sure your proposal would violate the spirit if not the letter of the BT protocol, but I'm out of ideas for how to convince you of that. All I can do is recommend you read up on the specific workings of BT and see if you can bring yourself to understand my point of view, whether or not you end up agreeing with it.

Link to comment
Share on other sites

Actually, defining how you must download a file is pretty much exactly what a protocol does.
Actually, the bittorrent protocol does not define the way how "you must download a file" (in this situation the order pieces are downloaded in). It just says:

"Downloaders generally download pieces in random order, which does a reasonably good job of keeping them from having a strict subset or superset of the pieces of any of their peers." [here], or

"Clients may choose to download pieces in random order.

A better strategy is to download pieces in rarest first order. The client can determine this by keeping the initial bitfield from each peer, and updating it with every have message. Then, the client can download the pieces that appear least frequently in these peer bitfields." [here].. It doesn't say anything about "you can't do otherwise" and "sequential" is a subset of "random". With this thinking, *.avi files f.e. should not contain mp3 audio tracks or mpeg4 files with b-frames either, because that bends the specification somewhat. Fools the specs rather :)

Finally,

Now, I'm personally pretty sure your proposal would violate the spirit if not the letter of the BT protocol, but I'm out of ideas for how to convince you of that.

If you're up for some more discussions, ask and i'll try to explain on what bothers you, because hey, the more believers the merrier ;) If not, then maybe you have a compromise (like the "hybrid" one offered earlier) feasible for both sides? If not that either, then let's just wait for the authors' "yay" or "nay", and i'll either step back, or rejoice, because a good client is about to get better ;)

Link to comment
Share on other sites

^^ The page you are referencing is not the BitTorrent protocol, it is a document explaining the protocol.

Okay, I thought of one last-ditch way of explaining: Imagine the simplest form of torrent: one person with a file, the seed, who wants to give that file to three other people, the peers. In this thread we have discussed three ways in which this could get done.

Seeding: The seed sends peers whatever chunks they need in random order. Some chunks may get sent to more than one peer, as chance dictates.

Super-seeding: The seed sends chunks to peers sequentially, with chunk A going to peer 1, chunk B going to peer 2, chunk C going to peer 3, chunk D going to peer 1 again, and so on around the circle until every chunk in the torrent has gone out to one peer or another. Meanwhile, as soon as peer 1 gets chunk A, he immediately starts giving it to peers 2 and 3, so by the time the seed has gotten through the alphabet, most of the letters have been successfully passed around and everyone's well on the way to completion.

Super-leeching: All three peers demand to download the chunks in order. Right off the bat, all three request chunk A at the same time. The seed obeys, spending three times as long as it would have taken to send it to peer 1 alone, before moving on to chunk B. Meanwhile, since all three peers are getting the same chunks at the same time, they have nothing to share with one another. In the end, the torrent data is distributed at a rate equal to the seed's upload speed divided by the number of peers connected. The pro is that if peer 4 comes in while peers 1-3 are getting chunk E, he can get chunks A-D from all three of them at once, giving him a very high speed right up until he's even with them. At this point, however -- and this is the big con -- he starts taking up yet another wad of the seed's bandwidth, and everyone's speed drops. If more peers keep joining in all through the alphabet, nobody will ever get to Z.

Although it is possible, as you say, to have a "compromise" in which chunks are handed out in sequential groupings (such as by file) instead of individually, the basic philosophy of peers getting to pick and choose is still out of keeping with the philosophy of BT, which is designed primarily to save the time and bandwidth of the seed.

Link to comment
Share on other sites

All three peers demand to download the chunks in order. Right off the bat, all three request chunk A at the same time. The seed obeys, spending three times as long as it would have taken to send it to peer 1 alone, before moving on to chunk B.
Super Seeder doesn't obey for one. It gives you a piece and you either agree with that and share that piece with others, or you get the hell out of the swarm. Super Seeder does not give the same piece again if it sees that piece in a swarm. As for a simple seeder... dunno, hybrid should be okay with that i think.

And when i mentioned "hybrid", i was talking about downloading one piece sequentially, the other "rarest first" (or 100 sequential, 100 rarest, no difference), so everyone would be happy :)

^^ The page you are referencing is not the BitTorrent protocol, it is a document explaining the protocol.

Well if it explains incorrectly, then it's a bad protocol reference and should be removed :) Care to point to the correct document? I thought bittorrent.com would be _the_ place to look for protocol specs..

Link to comment
Share on other sites

It's not incorrect, it's incomplete. And not all that informative, either, but it doesn't lie. I can't really tell you what's better, though, I've just kind of picked up what I know from numerous sources.

All I will say to your hybrids and your compromises is that you don't seem to be in much of a position to bargain. What you want is either legal or it's not, and no amount of dilution will make it otherwise. Unfortunately, this whole community is as decentralized as the torrents themselves, so even Bram can't tell you for sure what will fly and what won't. The closest thing to a final judge here is the tracker admins, who will each decide to ban a feature or not -- and it is up to the uTorrent devs whether they feel like risking that.

Link to comment
Share on other sites

The BitTorrent protocol says nothing about in which order the pieces are to be downloaded so any client can request/send pieces in any order chosen.

However the protocol is built for distributing random pieces to the peers to make it possible for every peer to achieve maximum download speed.

If however one client would download the pieces in a fixed order, the whole idea would be broken.

You know that already, but the request/send order of the pieces is _not_ included in the BitTorrent protocol.

Link to comment
Share on other sites

All I will say to your hybrids and your compromises is that you don't seem to be in much of a position to bargain.
Why not? A feature request is always a bargain :) And a compromise can be made if the other side is willing..
If however one client would download the pieces in a fixed order, the whole idea would be broken.
The idea of bittorrent is to decrease bandwidth consumption. It would still do that, even in the worst-case scenario (everyone uses it) you're all thinking about.

Just thought - how can one detect such a feature anyway? See whether a client requests sequentially, add noise; average position of pieces? use hybrid.. dunno..

Anyway, i suggest "let's just wait for the authors' "yay" or "nay", and i'll either step back, or rejoice, because a good client is about to get better ;)", because there is nothing left to comment on anymore..

P.S.: Senator Smartypants, you didn't point me to another document explaining bittorrent protocol, why should the ones quoted be wrong then?

Link to comment
Share on other sites

Right, that's a good way to put it. It doesn't say you can't send them in order, but the logical result of doing so goes against the goals of the protocol.

Edit: missed gribble's post there.

Gribble, seriously, I already told you that the document wasn't wrong, and just because there's not one single document that tells you everything about BitTorrent doesn't mean what I've said isn't true.

If you look at my "worst-case scenario", yes, it is better than HTTP in that incoming peers will catch up to the existing peers faster -- but that doesn't mean it isn't miles worse than regular BT. The beauty of the system we have is that a properly super-seeded torrent will take practically no more time to distribute than it would take to upload the same data to an FTP server. If the seed sends each chunk out to only one peer, that means they only need to spend enough total bandwidth to upload the entire file once. After that they can drop off and let the peers complete the file amongst themselves. If you start polluting this elegant system with a bunch of little "features" that break the protocol, you will end up practically right where you started, with an old-school decentralized P2P network.

As for trackers detecting the feature, they won't have to. All they'll have to do is detect the client, and since they'll have heard it had the feature, they will ban it regardless of configuration. And then all of us will suffer.

Link to comment
Share on other sites

The idea of bittorrent is to decrease bandwidth consumption. It would still do that, even in the worst-case scenario (everyone uses it) you're all thinking about.

Just thought - how can one detect such a feature anyway? See whether a client requests sequentially, add noise; average position of pieces? use hybrid.. dunno..

Anyway, i suggest "let's just wait for the authors' "yay" or "nay", and i'll either step back, or rejoice, because a good client is about to get better ;)", because there is nothing left to comment on anymore..

P.S.: Senator Smartypants, you didn't point me to another document explaining bittorrent protocol, why should the ones quoted be wrong then?

You say the idea of BitTorrent is to reduce bandwidth. That's true, in one aspect. It's main concept is to reduce server load so that one seeder can seed many more files at the same time. That is not aquired using sequencial downloading. You can read about BitTorrent on the Wikipedia (link). This image is pretty easy to understand too.

I will not sit and wait for the developers to decide. If they are implementing superleech, I will stay put with the version released just before they are doing so and hope that it will stay clean from bans by tracker owners.

Moving along to the detection:

It only takes _one_ peer reporting this "feature" to _one_ trackerowner and the word will be spread. (And the client will be banned)

Link to comment
Share on other sites

Gribble, seriously, I already told you that the document wasn't wrong, and just because there's not one single document that tells you everything about BitTorrent doesn't mean what I've said isn't true.
If it's not mentioned anywhere, then it's undefined, but you're stating here, that it's bad for the community, because it has a non-optimal worst case scenario (although it is very unlikely to happen). I say it's neutral to the community at worst, because the community is too big to catch on to one specific feature which doesn't really matter to most of the users. And noone considers the worst case prioritisation scenario for example..
The beauty of the system we have is that a properly super-seeded torrent will take practically no more time to distribute than it would take to upload the same data to an FTP server. If the seed sends each chunk out to only one peer, that means they only need to spend enough total bandwidth to upload the entire file once.
I explained that earlier. Did it state otherwise? Super Seeder commands the parade, not a peer..
It's main concept is to reduce server load so that one seeder can seed many more files at the same time. That is not aquired using sequencial downloading.
How is that? Could i hear the details behind your opinion?

And again, why do you think i suggest this as a default piece downloading algorithm? If you call it superleech it will definately attract attention from non-intelligent leechers and they will be downloading a file from start to finnish. Not faster, not more efficiently, just differently. Why would a leecher want that, could anyone explain? Yet, if you call it "sequential piece downloading", there will be nothing to attract those non-intelligent leechers.. That's the power of naming ;)

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...