Combining Torrents with identical FIles but different Hashes

[Qui Peccavit]

A situation that probably everybody has encountered a couple times:

 

Several torrents on different or the same sites/trackers contain the same .nfo, .exe and .iso files, etc., all identical in name, size and whathaveyou, with the only difference being some «uploaded by nitwit.txt» or «click this advertisement.htm» files. The irrelevant fillers alter the hash of the torrent, which means that BT client software such as μTorrent considers the otherwise identical contents as different and downloads them all redundantly in different folders.

 

This means that users are stuck with three or four different options, all of which are terribly slow, interrupted, cut off, whatsoever, resulting in much more hassle in form of copying between folders and forced re-checks, searching for alternatives, and so on, than necessary.

 

In http://forum.uTorrent.com/topic/8238-possible-to-have-multiple-torrents-saving-to-a-single-file/, there is an explanation that different torrent downloads trying to write the same chunks would result in «Access denied» errors and other havoc, because the BT protocol has not foreseen (although it is quite obvious, isn't it?) such a necessity.

 

The suggestion or feature request is therefore the following: if the protocol cannot (?) be enhanced and expanded to make it smarter, make its handling by the BT client software smarter:

 

When adding a new .torrent file to the client for downloading, there could be a simple checkbox «Combine with other Torrent» whenever the same name as an already existing Torrent is entered as the destination. When μTorrent encounters the same file names in the new torrent as in the exisiting directory, it would automatically compare file sizes and, if those are the same, too, do everything else automatically (we will come to what that is in a moment). If the file names are not identical, μTorrent would present a screen with a left and a right window, in which users can move up/down file names to indicate which file in the additional torrent is the equivalent to which file in the existing one. Of course, a smart software would compare file sizes and sort matching file sizes automatically, but the user should still have the opportunity to correct the suggested order.

 

Once the user clicked «done», «combine» or «integrate the Torrents now», μTorrent would begin to analyze how many seeders each of the torrents has and assign the pieces accordingly. For instance, if the same content (apart from irrelevant .txt files or the like) is available in three different torrents -- whether from the same or different sites/trackers is irrelevant -- say, A = 5 seeds, B = 3 seeds, C = 2 seeds, then 50% of the chunks/parts/pieces would be assigned to be downloaded from tracker A, 30% from B and 20% from C. The user could do all this manually, by downloading three different torrents into three different subdirectories, setting the other 50%/70%/80% to «do not download» and copying everything into a single directory in the end, but the whole point of computers and their raison d'être lies in automatizing tedious manual tasks. Moreover, the assignment is not to be seen as static, but to be re-evaluated and re-calculated in certain intervals. When all the parts have been downloaded from tracker B while A and C are still busy, the latter should be relieved of some of their burden and more chunks should be assigned to B. By the same token, if 3 of the 5 seeds on A turn their machines off over night, the weights need to be re-balanced as well. Taking into account other factors, such as the speed at which each seeder shovels up the data to a client or how often the connection is interrupted in order to seed to other peers, should make for some pretty smart AI that results in the best possible availability and speed for old or not-so-mainstream torrents.

 

This way, the tedious searches for alternative torrents (when seeds suddenly disappear), manual file copy and recheck operations as well as painstakingly slow speeds on each of several torrents containing the same files would be a thing of the past.

 

I may not have thought of every detail right now, this was more or less off the top of my hat after reading the aforementioned thread that one can no longer reply to, but where there is a will, there is surely a way. Seeing how many people asked about this topic over the years should make its consideration and implementation worthwhile.

 

Thank you for your attention and have a pleasant day.

 

 

Yours faithfully,

QUI PECCAVIT

[DreadWingKnight]

Duplicate request. Because of the way the torrents are structured, it's far from trivial to get the torrents to combine.

The protocol doesn't have a concept of files at a level where it would take the desired amount of work to get the torrents to actually combine.