bigger torrents

[Anoid]

what can be done to fix the torrent protocol to support larger torrents?

is there any reason that a torrent cannot be made to support 100TB or even 18EB of files?

I realize 99.99% of torrents do not need this much big data, but I can think of many.

[MisterRemote]

I can't think of any single torrent that would need to be such a size. Torrents are for mass distributing, show me more than 10 torrent users worldwide which would even have that much of disk space to download it and a fast enough internet connection to share it within a year. Really there is no practical usage for this for the next 20 years at least.

[Anoid]

I can't think of any single torrent that would need to be such a size. Torrents are for mass distributing, show me more than 10 torrent users worldwide which would even have that much of disk space to download it and a fast enough internet connection to share it within a year. Really there is no practical usage for this for the next 20 years at least.

I can think of dozens of ideas, computer chess, astronomy and more.

there is more to torrents than an MP3 or AVI file

[ciaobaby]

The people and organisations that would need such transfer are smart enough to figure out ways other than BitTorrent.

http://www.cs.unh.edu/~varki/publication/PDCS2011.pdf

[vze2mp9g]

The people and organisations that would need such transfer are smart enough to figure out ways other than BitTorrent.

http://www.cs.unh.edu/~varki/publication/PDCS2011.pdf

I totally agree with this statement. You really haven't been around the torrent scene lately because I have never seen a torrent 1TB in size, even if you decided to do an ISO of a Blue Ray disk, that would only make the torrent approximately 50Gb!

Let say you did make a torrent that was 1TB, who in their right mind will attempt to download it? I see people complain when the torrent is over 20Gb. It's best to keep the torrents as small as possible because most people use their computer for other things as well, and if you loose your power supply due to a storm, it would take a while to check the data and resume downloading.

Now, if you were a business, then it's a totally different story, and I would think that they don't use BitTorrent, and they have a lot faster IntraInet connection.

Like someone mentioned about, we (the common folk) won't need something like that until 20 years down the line.

[ktetch]

I can't think of any single torrent that would need to be such a size. Torrents are for mass distributing' date=' show me more than 10 torrent users worldwide which would even have that much of disk space to download it and a fast enough internet connection to share it within a year. Really there is no practical usage for this for the next 20 years at least.[/quote']

I can think of dozens of ideas, computer chess, astronomy and more.

there is more to torrents than an MP3 or AVI file

One of the other things I do, is work on a distributed computing project designing particle accelerators. The computer designs one, runs a simulation by running 80,000 particles through it, in 0.01nanosecond steps, and sees how it does. It then works from that design and ALL PAST DESIGNS to create the next one, working evolutionary, not brute force.

10 years of this work, 80 million designs, simulating to the point where you can simulate a particle from here to pluto and back 5 times over, in 3mm steps, and all the data fits on a single terabyte hdd. Mainly because a single design and result is only 1kb.

Massive data just doesn't work for these projects (same as radio astronomy either deals with the data there and then, on site, or breaks things into very small packets. And discards what's useless)

Your 'need' has no need.

[Anoid]

http://computer-chess.azurewebsites.net/download.php

 

the torrent for 6 pieces was possible thanks to the team who noticed a defect with larger sets

 

those involved historically used RAID but today a single disk is now able to hold the 6 pieces set

 

Chess programs are free and there are others that are commercial. My site provides the free stuff, the EGTB work with chess engines to augment the endgame which is hard to program.

[Anoid]

I posted a new torrent on the chess site, I decompressed the set, then used 7-zip to crush it down more. The compression used is designed to be opened while remaining compressed.

 

EMD can achieve 4x compress of EGTB, 7-zip is better but it has to be unpacked to be used. There is overhead for the EMD compression, none for the NBB and NBW raw files.

 

The NBB and NBW files are indexed into a table called distance to mate, the index is made from a hash of a given chess position.

 

A lot of hard work has gone into EGTB, and Microsoft provided a server for the computation, a 4 socket Alpha server.

[Anoid]

here is a backgrounder on computer chess where the end game work started all the way back to the 1970s and earlier

 

http://computer-chess.azurewebsites.net/wp/egtb.php

 

Computer chess is so extreme it literally attracts some of the brightest to work on the problems

[Anoid]

I have updated my guide for EGTB distribution. Using a torrent is so brutal that it takes over 2 years with consumer internet connections.

http://computer-chess.azurewebsites.net/wp/chess-torrents.php

[Anoid]

I noticed a lot of gripes over the size of some recent game torrents. Games are getting a lot bigger now/