Getting a stuck torrent going again

[cjard]

I've a large torrent stuck at 96% and I've come to realise that maybe it could be rescued if the same files are present in another torrent. I'm downloading another that seems to have the same files, albeit different names.. I think I can stop the broken one, replace the part downloaded files with their equivalents from the other torrent, force a recheck and get it going again?

If so.. this got me thinking -> could uTorrent do this? When a torrent gets stuck, right click it and choose to patch it with files from another torrent.. Map a stuck file to one from the other torrent, anf then uTorrent could download sufficient pieces to assemble the missing piece from the stuck torrent?

[ciaobaby]

replace the part downloaded files with their equivalents from the other torrent, force a recheck and get it going again?

No.

If so.. this got me thinking -> could uTorrent do this?

No.

[cjard]

I beg to differ; it worked with the files I tested.. I need to read up more on the bittorrent protocol, but I don't see why it couldnt be implemented that a BT client witha stuck torrent could be told "here's another torrent that might contain the same files" and it would then work out which pieces it would need to load from that torrent in order to have the relevant data to reconstruct a piece from this broken torrent. I never claimed the piece boundaries fell in the same place, mind!

[ciaobaby]

If they are EXACT BINARY equivalents then it is theoretically possible, HOWEVER the chances are more, that you will simply corrupt the torrent and start causing hashfails for any other peers that are downloading from your client.

For it to work 100% successfully the hashsums of every piece MUST to match the metadata AND the files have to be across EXACTLY the same piece boundaries, otherwise the pieces of the jigsaw will be going back together in the wrong order.

It's a very interesting end result when that happens on MP3s and two torrents somehow get put together in the wrong order.

[lovetour]

If they are EXACT BINARY equivalents then it is theoretically possible, HOWEVER the chances are more, that you will simply corrupt the torrent and start causing hashfails for any other peers that are downloading from your client.

For it to work 100% successfully the hashsums of every piece MUST to match the metadata AND the files have to be across EXACTLY the same piece boundaries, otherwise the pieces of the jigsaw will be going back together in the wrong order.

It's a very interesting end result when that happens on MP3s and two torrents somehow get put together in the wrong order.

It is literally possible even with different piece boundaries (which there will nearly always be with different torrents).

uTorrent does not support having multiple torrents downloading to the same files from totally diffferent torrents, even if the files are the same; at least not at the same time. Seeding using multiple torrents seems to work fine.

To insure you don't introduce hashfails like ciaobaby warned, is:

(1) if you download from multiple torrents to the same files (and your sure the files are exactly the same), you must only have one torrent active at a time, and have the others stopped. Before starting one of the others, stop the current one, and do a Force ReCheck on the torrent you are starting; that will get it to correctly verify the pieces as that torrent sees them against it's hashtable and prevent any bad data transmission or redownloading of parts of the file you've already downloaded. A mostly minor downside is that you will lose any partial pieces previously downloaded but not finished and those pieces will be redownloaded in their entirety (unless already downloaded using another one of the related torrents). It would be SUPER DUPER awesome if utorrent added a feature to download simultaneously from multiple torrents!! It could coordinate the overlapping hashtables and such itself, allow simultaneous downloading, not throw away partial pieces, avoid human error from the complex manual method I'm describing, etc.

(2) If you have all the files entirely, and just want to seed via multiple torrents that have different infohashes but identical files (or even if some torrents contain many files and others only contain some of the same files), simply download the .torrent files for all the torrents you want to use, and point them at the same location before starting the torrents, and do a Force ReCheck before starting the torrents to verify that they really do match up binary-wise. Then start them and they will seed compatibly in my experience.

(3) Beware of a "Gotcha" in all this -- if you use the option to have uTorrent name partially downloaded files with the .!ut extension -- it is very easy to confuse uTorrent unless you understand what's going on during the whole process here exactly and have uTorrent create two copies of the files - ione with !ut and one without. For downloading from multiple different torrents (like I said, only one at a time though), it's best to disable the .!ut option, because each torrent may have a different view of whether a file is complete or not, particularly considering differing overlapping pieces that span two files. You don't want uTorrent to create multiple files for the same file, because they cannot be combined, so best to set uTorrent to just use one filename from start to finish no matter what. For seeding, it shouldn't matter, but stop all torrents using those files when you do a Force ReCheck for any torrent acting on those files if you use the .!ut option, as the files get renamed to !ut and back during the ReCheck and it may mess up the state of the other torrents.

LT

edit: I have successfully downloaded to the same set of files from multiple different torrents successfully before - but it is a bit hairy so I don't reccommend it unless you really know what you're doing. The key here is that no matter how you do it, you don't want uTorrent to try to write to the same files on disk for more than one torrent at a time. SO, to achieve this, you need to select different files for download from each torrent (setting other files to "do not download", and BE AWARE that even files set to "do not download" wil be downloaded in part if the are in the list of files (in part# order) for a torrent and come right before or right after a file you selected for download (since there is nearly always an overlapping piece that is in both files). You have two options for dealing with the overlapping piece issue: (A) disable download of all files that border selected files, keeping a "gap" of two files for simultaneous download (so each torrent can have access to it's own de-selected file that has just that partial overlap piece it needs) ( enable the option to use a partfile in the uTorrent options (I think it's an advanced option). This way instead of creating a file with the usual filename for the de-selected files with an overlapping piece, uTorrent will create one file for each torrent, called something like ~partfile~123416535 or somesuch that will contain all the overlapping parts of pieces not in selected files. This option makes so you can select ALL files for download in one or another torrent and not worry about overlapping pieces causing file-access problems. Whenever you change which torent is downloading which files, be very careful that any other torrent already accessing those files is stopped as you change things around, and if you are taking over files-download from one torrent to another, do a Force ReCheck before resuming the torrents. Tricky.. and people doing this sort of thing are likely the people who make mistakes in the doing of it and cause hashfails on the bittorrent network to begin with...

But, I reiterate, it is possible, if you are careful and capable enough.

[cjard]

Indeed, this is why I started this dsicussion in the feature requests forum, as I think it would be possible to add it in as a feature, and probably a fairly unique one at that

Essentially, I've repaired my stuck torrent now; I downloaded files from another torrent, some with completely different names which did make things a bit of a pain int he ass. I then stopped the "good" torrent and moved those files over the top of their relevant !UT equivalent in the stuck torrent, did a force recheck and it all seems good. The logic of why this works is obvious; utorrent doesnt know, or care between stop and starts, where the data came from for a file

The downside of this approach? For example there was one 700mb file that was missing just 4mb in one piece. I had to download the entire file again in the good torrent, which is a huge waste of bandwidth, but because I (as a user operating at the OS/file level) just didn't know which 4294967296 bytes i needed, and even if I did, I would have no way of telling uT to grab just those or the pieces that contain those, from the good torrent

This is where the feature request comes in

Suppose a torrent gets stuck. It contains 100 files of 100 mb each and the piece size is 4mb

One file is missing one 4mb piece number 3159, lets assume it's file79.avi and it's missing data from byte offset 12341234 to 4307308530

The user right clicks the file and chooses the "Source this file from another torrent" option

uT asks for a torrent file to be loaded and the user gives it one

uT loads the torrent and finds multiple files (based on bytes size) that could be the candidate file, no filename match is found either

uT asks the user "which file in this new torrent, is the equivalent file?"

user picks file_seventy_nine.avi in the new torrent

Now suppose the new torrent is using 3mb pieces .. sure, I can accept that the piece boundaries wont fall in the right place, so some pieces will have to be got and thrown away.. uT already does this if you skip all the files in a torrent except one - the pieces that overlap the edges have partfile data that is discardable

so uT knows it needs bytes 12341234 to 4307308530 of file_seventy_nine.avi to patch into the file79.avi.!ut already on disk

let's assume uT works out it needs 512kb from piece X, 3mb from piece X+1 and 512kb from piece X+2

uT downloads 9mb of data by way of piece numbers 5712, 5713, 5714 in the new torrent, clips out the data it wants, hashes the concatenated resulting 4mb and if it matches, it inserts it into the file...

Thus we repaired our 700mb file by downloading just an extra 9mb

[lovetour]

There is a Bittorrent client that introduced a unique feature some years ago, Shareaza, to simultaneously download the same files from multiple P2P network simultaneously (Bittorrent, EDonkey2000, Gnutella, and Gnutella2). I would not be in the least surprised if it handles multiple sources on the same network as well, including Bittorrent. Although I don't know. The other networks hash shared files on an individual basis making locating specific files easier than on Bittorrent. With Bittorrent, it would seem to require user intervention to suggest torrents that share the same files.

Shareaza is open source and had significant development and a significant userbase until their domain was apparently seized, and started hosting a fake client from a copyright consortium, which it still hosts to this day. The real client moved to Sourceforge (http://shareaza.sourceforge.net/) and is still being developed, although it would appear that development has slowed over the years.

Since Shareaza is open source, and uTorrent is too, uTorrent can probably even use or adapt code developed for Shareaza (depending on how each is licensed, which I don't know).

The downside of this approach? For example there was one 700mb file that was missing just 4mb in one piece. I had to download the entire file again in the good torrent, which is a huge waste of bandwidth, but because I (as a user operating at the OS/file level) just didn't know which 4294967296 bytes i needed, and even if I did, I would have no way of telling uT to grab just those or the pieces that contain those, from the good torrent

Actually, you didn't have to download that file again. You could have used uTorrent's file>Relocate feature to point the second torrent's file at the first torrent's file (even if the filenames were different), made sure the first torrent won't access that file as I previously discussed (or stopped the first torrent), done a Force Recheck on the second torrent, and then started the second torrent, and it would only download what it needed to to complete the file.

However, it is a feature that would benefit plenty of advanced users who do all kinds of juggling like I've suggested currently.

An extension to the bittorrent protocol itself that shared hashes of files as a whole -- well, that would make automation of linking associated torrents a breeze! If users added a second torrent that had some or all of the same files, uTorrent could then automatically use both "swarms" for those files. But even without that extension, a way to group torrents and files would be great, where uTorrent can "take our word for it" at least until it downloads enough pieces from both torrents to know that files we've specified as the same actually aren't. To play it safe without the Bittorrent extension I mentioned, uTorrent could mark pieces complete for a given torrent only when the a piece can be hash checked against that torrents hash table (might be parts of two pieces from another torrents hash table), and if a difference between the files for the two torrents is found, stop the torrents with an error, or exclude those files with an error, or make a second copy of the file so each torrent downloads its unique version -- with an error..

LT

edit: correction: uTorrent is not open source, sorry for any confusion.

[lovetour]

In terms of a feature request, I notice two things; .

(1) .torrent files currently have an optional component of an MD5 hash for each file included -- when available, that could be used to identify shared files. MD5 is not used by the bittorrent protocol, and I don't know how common or uncommon their inclusion in torrent files is, or if uTorrent generates .torrent files with MD5 hashes or not.

(2) The BEP 38 (Bittorrent Enhancement Proposal 38) - http://bittorrent.org/beps/bep_0038.html - titled "Finding Local Data Via Torrent File Hints" has many of the same goals and introduces ways that torrent files can specify other torrents with shared files, ways that torrent clients can automatically identify shared files when they come across them, and ways that torrent clients can even use data from other torrents that is the same at a resolution smaller than a file (my example: a media file may have different headers in different torrents but the primary data is the same; BEP 38 includes a proposed method for clients to automatically identify such cases when possible).

The title of the proposal refers to torrent file hints. Specifically the hints include a possible list of Info Hashes of other torrents with shared data and Collections to which the torrent belongs (i.e. a TV episode is a member of a TV series).

BEP 38 appears to offer some ideas on how the problem might be approached, although it does not discuss a whole system for managing multiple sources in detail.