create torrent files added are sorted by modified time

[eeipcy]

Hi,

I use a lot of sites that issue updates to torrents often, they use torrentzip to make sure that the hashes of zip files are consistant across platforms.

One major problem with using torrents to update many files is that the latest files added are never in contagious blocks. Therefore people end up downloading large amounts of data that they already have, needlessly.

One way to solve this would be to index files based on modified time, so that the newest files added would be in a contagious block at the end of the torrent file. (The other boring way would be to rename all the files so that all newly added files are in numerical/alphabetical order, but this is not always possible.)

A new tickbox option when creating .torrent files in uTorrent would be nice, so that it would add files in to the torrent file in order of their last modified time.

Thanks in advance,

eeipcy

P.S. This is not my idea, but I do think that it is a brilliant idea.

[jewelisheaven]

uhm, the modified time is not stored in the torrent. It was a relatively early change in 1.7 which brought about the "always sort by name" in torrent creation. Previously it was created in-order-of selection. Which could have been any way (I don't actually know what the previous method was) for multi-file torrents.

Interesting idea, indeed, however unless there's many more types of ways to create added, it seems like too much effort for such a niche market.

[Firon]

I don't see this happening.

[eeipcy]

One last post on the subject, by me, before I let sleeping dogs lie.

uhm, the modified time is not stored in the torrent.

That is not what I am saying, modified time is not stored in the .torrent, the modified time is only used to select the order in which the files are added to the .torrent file.

From the source code of torrent creation programs that I have read so far there are only two ways in use today.

1> Scan through directory structures and add files in the order returned by readdir() to the .torrent file.

2> Scan through directory structures, sort the found filenames and add to .torrent file in alphabetical order.

What is being suggested is a third way. When reading the filenames and filesizes into a link list, add an additional field for the modified times of the files (probably in POSIX time format, it would make the most sense). So that when new files are inserted into the link list, that they are added in order modified time. Oldest to newest, this order of files is then uses as the order to put files into the .torrent file. The modified time is never stored in the .torrent file.

And the reason for this is to move all newly added data into a contagious block at the end of the torrent. So that when people go to update a torrent to the latest revision of that particular torrent. That they will only ever have to download the newly added content and one additional "Piece size" due to overlap.

That they can update their previously downloaded torrent, without having to download a lot of data that they already have due to "Piece size" overlaps distributed throughout the torrents data.

==============================================

Just to be totally clear that you get what is being suggested:

Old torrent data: AFQY

(data added in first of many updates: SZBCH)

Current torrent data updates: ABCFHQSYZ

Due to "Piece size" overlaps I would have to download parts of AFQY again!! (Or possibly the entire files again if they were smaller than the Piece size and fell poorly on the torrent piece boundaries)

Suggested torrent data updates: AFQYSZBCH

I would only have to download part of Y again, because the newly added data is added to the end of the torrent.

(data added in second of many updates, days/weeks/months later: DGJORU)

A traditional .torrent created, sorted alphabetically, would be: ABCDFGHJOQRSUYZ

Because of piece size overlaps bits of C, F, H, Q, S and Y have to be downloaded (again) in addition to the update files: DGJORU

But if a file timestamp ordered .torrent were created instead the update would be: AFQYSZBCHDGJORU.

This also means that if someone missed, one or more, updates that the amount of data that they would have to re-download would not be continually increasing.

[Firon]

all keys are supposed to be sorted as raw strings, so really, this would go against spec recommendations

[eeipcy]

So it looks like the only way that this will happen is if the .torrent metainfo is extended to support file ordering. Which would mean that two sets of piece hashes need to be stored to maintain backward compatability. Which would make the .torrent files almost double their current size. *sigh*

Thanks Firon.

[Huggybaby]

Well, assuming that it could be made possible...

There are a great many users for whom this would be a welcome feature, so I don't see it as a "niche" feature at all. And it would reduce a lot of bandwidth in common situations, especially when updating any collection of files. And who can argue with more efficient use of the limited resource that is bandwidth?

I've been using 1.8 beta off and on, and following the progress thread, and it seems that the issues hanging up the new release would not be missed by me were they removed completely. For example, I don't use UPnP, or RSS, and I wonder how many torrent users do. I don't even use the search feature, I use Google for that.

But this OTOH sounds like exactly the kind of tweak that suits µTorrent best, current spec be damned. Such an obvious idea is guaranteed to be implemented by someone, µTorrent or otherwise, once the shock of It Can't Be Done wears off, because it happens with any new idea worth its salt.

After all, Bittorrent had no problem effecting DNA, and that must have been a more involved undertaking than what is suggested by implementing this feature.

Huggybaby

[jewelisheaven]

Yup it is a tweak. It is not in the spec, and as the purveyors of the protocol it would be bad form for them to do so. Check out wiki.theory.org , there are MANY "additional features" not implemented by uT which for ex. Azureus has due to this limitation.