DHT : what if the bootstrap dies?

[EdX]

Hi,

this is my first message on this forum. I hope my questions have not been answered yet. I searched and could not find any information, I swear!

I invented the following story: John creates a torrent without a tracker. So he is the DHT bootstrap. When Jack adds this torrent to his bittorrent client, he connects to the bootstrap (John) and start downloading. When Paul adds this torrent to his bittorrent client, he connects to the bootstrap (John), and John gives him the address of Jack so he can start downloading from two sources. Etc, etc…

Am I correct with this little story, or did I misunderstand something?

Now what happens if the bootstrap (John) dies? I think Jack and Paul can continue to exchange data. Am I still correct? But what about new comers? Can they reach Jack or Paul?

[Ultima]

The creator isn't the bootstrap; the default bootstraps are located at router.utorrent.com and router.bittorrent.com. For a user to add himself as the bootstrap for the torrent, he needs to modify the torrent to include a `nodes' list in the root dictionary where each element in the list is an IP:port combination that acts as a DHT bootstrap.

If the default bootstraps for DHT die, then you can add to µTorrent's torrent jobs list a torrent with a `nodes' list that contains known-to-be-working bootstraps. µTorrent will use those nodes as DHT bootstraps.

[EdX]

Ok, thanks for you reply.

So it means that even if we don't want to use a tracker, we are still using a central server (which is router.utorrent.com).

I thought that the aim of the DHT functionnality was to get rid of that kind of constraint.

How can I edit my torrent file?

[Ultima]

Sure, the aim of DHT is to further decentralize BitTorrent, but if there is no entry point, how would the client discover nodes on DHT? There has to be some known entry point, and that's what router.utorrent.com serves as.

The bootstrap isn't a constraint; otherwise, it wouldn't be a "bootstrap" -- it would be the centralized server. If the entry point goes down, DHT wouldn't go down -- you simply need to find another entry point and get on DHT through there, and that's what the `nodes' key would help µTorrent do. The same can't be said about a traditional tracker whose server goes down, and that's why DHT is more robust (it has no single point of failure).

http://bittorrent.org/beps/bep_0005.html#torrent-file-extensions

You can edit .torrent files using BEncode Editor.

[EdX]

Oh I see, so router.utorrent.com has to be seen as one of the entry point, but not as the only one. But when I create my torrent file, is my IP address also added as an entry point?

Given what you just said, if I know another entry point (my computer at work, my computer at home, …), I should also add it, shouldn't I?

[Ultima]

µTorrent doesn't create a nodes key automatically, no. I'm not sure it's really a question of "should." Unless you have a real reason to believe router.utorrent.com will be inaccessible soon-ish, you don't need to need to manually add your own clients as bootstraps.

Yes you can, though, if that's what you're getting at. Oh, and the entries in the nodes key don't need to have IPs -- it can contain domain names (so if you don't have a static IP, you can still use your own computers as bootstraps if they have domain names).

[EdX]

Ok it's much clearer now.

I actually have no real reason to believe router.utorrent.com is going to be inaccessible. But I am interested in how the bittorrent system, and p2p systems in general, could work without any central point.

The internet world today works thanks to a "few" big central points : google, wikipedia, trackers…

What's gonna happen to us if they stop working? Using trackerless p2p (bittorrent, faroo search engine…) makes me feel less powerless about that

Thanks again

[hermanm]

If you take TPB as an example... makes you wonder what kind of distributed system could take on the workload of a tracker that size (download .torrents, search functionality, tracker updates) and still perform at an acceptable level. I had to turn off DHT because it tends to kill my router, which would suggest other individuals do not have the hardware and/or network connectivity to handle such a task?

Someone needs to write a tracker that runs on Google App!

[EdX]

hermanm, I agree. Distributed systems can't be better than centralized ones in every way. But in my opinion distributed systems has two main advantages :

* They are less likely to die.

* They create competition to fight monopoly (google, ...). I says them : "Careful men, we can live without you"

For now centralized systems work well, but I think it's a good thing to keep using ditributed ones at the same time.

Not to mention the costs generated by servers (or even server farms!).

[Switeck]

DHT isn't nearly as robust without Peer Exchange also being active.

In fact, if most everyone uses Peer Exchange then it would only take a few also using DHT to knit themselves back together when/if the centralized tracker/s go down.

[jch]

One thing to understand is that there's only one DHT in the universe, not one DHT per torrent. The way this works is that if there are two DHTs, then as soon as a single node is on both DHTs then the two will be merged.

The only issue that bootstrapping needs to solve is how to join the DHT in the first place. In order to do that, you need to find a single node that is already in the DHT (not one per torrent, just one in all). Most DHT implementations, including the one in µTorrent, will use multiple methods to find that initial single node:

- a list of nodes saved when you last shut down your Bittorrent implementation;

- whenever you contact a new Bittorrent peer, if it is on the DHT, it will tell you so (that's the PORT message);

- a torrent file can include a set of bootstrap nodes.

I believe that µTorrent also includes other, non-standard methods to find DHT nodes. Perhaps a µTorrent developer can enlighten us?

If all else fails, and only then, will you rely on the hard-wired bootstrap nodes (router.whatever.com).

--Juliusz

[hermanm]

Is DHT & Peer Exchange just as effective as a tracker for retrieving a peers list? For example, can I host a trackerless .torrent on a web host and expect to find all peers?

[DreadWingKnight]

Until/unless Azureus/Vuse puts their secondary DHT plugin as part of the default install (the mainline DHT support) then you won't get access to all peers until someone with that plugin installed joins.

[jch]

> Is DHT & Peer Exchange just as effective as a tracker for retrieving a peers list?

Taken together, they're pretty good. DHT gives you a reasonable number of randomly spread peers, and PEX allows you to find neighbours of the peers you got from DHT.

There are a few reasons to stick to trackers when possible, however. First of all, trackers are faster -- a tracker gives you 50 peers in a few seconds, while the DHT takes half an hour to get up to speed, and a few minutes to complete an announce. So a tracker will allow you to find peers as soon as you start a torrent, while using the DHT means that you need to wait.

Furthermore, many DHT implementations are buggy (my DHT implementation receives impressive numbers of bogus messages, and I see a lot of things in the DHT at places where they shouldn't be -- it looks like some clients don't correctly implement searching for the right nodes to announce to).

Finally, as DreadWingKnight notes, many implementations implement a non-standard version of the DHT, or don't implement the DHT at all, so tracker-less torrents won't be visible to them. (This includes at least Vuze/Azureus, which uses a non-standard implementation, and transmission, which doesn't have a DHT at all; thankfully KTorrent implements the DHT now.)

--Juliusz

[streetstopper]

in regards to ultima's post, and for clarification, should "nodes" be along info, announce, comment, etc?

i have downloaded a torrent file that includes "nodes", created by a bitcomet, inside "info", thus changing the torrent's hash when changing/adding/deleting nodes URLs.

further, how should the structure of "nodes" look like?

thanks guys for helping.

[DreadWingKnight]

i have downloaded a torrent file that includes "nodes", created by a bitcomet, inside "info", thus changing the torrent's hash when changing/adding/deleting nodes URLs.

That would be BitComet misbehaving AGAIN.

An example of a functioning and correct "nodes" key can be found from the openoffice torrents.

[streetstopper]

thanks DWK for your quick replies to my querries. i am in the process of improving my torrent file maker and adding more functionality but at the same time, wanting to stick to the standards.

will check openoffice.

thanks again.

[Ultima]

in regards to ultima's post, and for clarification, should "nodes" be along info, announce, comment, etc?

... include a `nodes' list in the root dictionary ... [ED: emphasis added]