Improving Traffic Locality via Biased Neighbor Selection

[rafi]

@booky: do you really think you can get meaningful number of peers to achieve proper speed for one of YOUR torrents ONLY from among peers from your country ? in the past I also suggested peer preference based on the distance (=ping/ip/country "flag") but seems to me that the idea was not very popular. You might want to look at Azureus plug-ins. I remember mentioning of such implementation there.

[booky]

@Ultima: that is interesting option, but the thing here is a bit different. I use 2 utorrent instances. One for internet torrent trackers and the other one for torrent trackers from my providers ip range. Using 2 utorrents allow me to know how much bandwidth i have left. In my case priority has utorrent used for internet. So, i can't use ipfilter since the idea was to speed up download from internet. To try to get local peers come faster from internet trackers.

@rafi: I believe i could get few peers from my local browiders ip range. For eg. when u have 1Mb + 1Mb for local peers it is double speed. There are people who are not using this bandwidth for local peers at all so it might be beneficial. People were telling me about them downloading some torrents where they found by chance other local peers. It was way faster.

Thanks for prompt answer

[patcat88]

The actual solution for P4P-like system is extremely easy. Inter-ISP links are almost always defined using ASN numbers/networks. Have BT client do a ASN lookup on its IP and get that ASN's IP ranges. Unchoke peers whose external IPs fall into that range first. If all the same ASN peers don't have pieces, or too slow, or whatever, go to non same ASN peers. Traffic between 2 IPs in the same ASN will never leave that ISP's equipment/network.

Here is the ASN IP ranges for one of the largest ISPs in the USA.

http://fixedorbit.com/cgi-bin/cgirange.exe?ASN=7922

[rafi]

I also like that, but will it not result in a too big overhead (for very limited benefit) ? there are thousands of ISPs in the world some are small. checking ALL connected IPs to get a very small percent positive result sounds ineffective. At the time I suggested country-based proximity detection, since flags/country codes are currently checked in uT so the info is already available. Though ping-based selection would be best.

If ping based proximity is a bit difficult to do, I would go -

A. preference to ISP local net

B. then, preference by local country (that may also save some people money ...)

C. then, check for closest countries

The user may be able to select to force it to do only some of the above

more notes are in here: http://forum.utorrent.com/viewtopic.php?pid=317283#p317283

[CrazyDiamond]

I wonder if it would be simpler if ISP named trackers were set up, and 'promoted' in Google so that users of a specific ISP would go to that tracker. For example:

isp1.abctracker.com

isp2.abctracker.com

isp3.abctracker.com

isp4.abctracker.com

Users of ISP 1 would go to isp1.abctracker.com to fetch their torrents, and users of ISP 2 would go to isp2.abctracker.com to fetch theirs. Then you know you have a high number of seeds/peers who are more local.

Challenge with this is, the swarm is likely to be smaller, me thinks.

[DreadWingKnight]

And how would data get across trackers to different ISPs?

[CrazyDiamond]

"There is nothing stopping a user from ISP1 from fetching a torrent from ISP2."

----Edit---

Let me rephrase that. There is nothing stopping a user with ISP1 from fetching a torrent on a tracker, labeled for a different ISP....ISP2 for example.

[Firon]

Kinda defeats the purpose.

[CrazyDiamond]

...but it does allow users to peer up with those on the same ISP which would be a very simple way to accomplish keeping some traffic more local....would also be a very simple way to prove or disprove the debating going on in here without having to muck around with the protocol spec.

[Switeck]

And if one ISP's tracker is nearly empty, they will flock to other trackers...thus making them "poisoned" with various ISP ips. Your scenario would only stay "pure" until someone sees the need to jump trackers...and then any benefits it might offer to "local peers" becomes moot.

[CrazyDiamond]

I kinda figured that when I was thinking of challenges with smaller swarms.

[atlantisisdead]

I never had a serious problem with slow downloads. Or uploading. I always upload much more than a fair share. However, recently with 3 of my last few torrents, some downloads have been extremely slow (.05kbps or 1.2kbps up to maybe 3.1kbps and back down again) while uploads remained normal. Usually my total download speed is up to 70kbps (normally around 35-40, as it obviously depends on mutual speeds) split up, of course, between peers on the list. Same with uploading.

Here's the kicker - two of the torrents had 2-4 seeders who were apparently uploading to other peers, but not to me. Ever. My download came from peers only.. as they got it from others. Further, on these 3 torrents, faster speeds (kind of) were from new peers only, but once they reached a share of 10% or so, it dropped to .05 kbps, 1.7 kbps, 2.03 kbps, etc. Every time, for every new peer joining. I watched other peers get theirs and go, while those same 2-4 seeders seeded everyone but me. I have a great ratio. If they targeted me for some reason, I wouldn't know why.

And it was only for these 3 torrents. Now, I read ultima's post about selecting peers by speed:

"Because BitTorrent doesn't work that way, and doesn't need to. If you upload more, more users will recognize that you're doing your part in contributing and should (in return) prefer to upload to you rather than peers that don't give them as much data. That's why uploading is important in BitTorrent."

And I am left wondering, how can peers and seeders target someone? I can see where peers can be removed. I found where total speed for a torrent can be reduced or increased effecting all peers. But I cannot see where certain peers can be given lower or higher speed rates when I upload to them. How were they able to reduce their upload to me? The 2-4 seeders may have removed me from their peer lists, but the huge number of peers all joined together and reduced their upload speeds to me? I use uTorrent, not BitTorrent. Does BitTorrent allow people to do that?

-----------------------------

It's now 9/09/09

Well, I'm still tryin' gto figure it out. Other torrents are fine. I haven't done any searches on BitTorrent yet to find out what it is if it is, indeed, something different from uTorrent. Still trying to figure out how those particular seeders on those particular torrents are able to keep me from downloading to my full speed by not uploading anything to me themselves and limiting the speed of the peers who are not yet full seeds. The peers will upload to me quickly only until they have reached a 10% (approx) total ratio and then the speeds slow down to practically nothing. All other torrents upload and download normally, including the part about everyone leaving when they get theirs leaving me to stick around for everyone else as the only seed. Heh heh. Hey, wait, that's not funny.

Anyway, still wondering how they can control it like that and why they are targeting me, if so, I have no idea.

------

well, it wouldn't let me make a new post - forced me to add to this one. I think it's gonna end up being a really really really really long post. Anyway, I don't remember the actual date of the original post. It replaced it with today's date. I think it was September 3rd.

Pretty sure it was 09/03/09....

[Lazybones]

Many ISPs shape there outbound and inbound traffic to the net but not their internal traffic as much.

For example I can get more than TWICE the download speed between me and my friend on my home connection on the same ISP than if I use a link technically 3 times faster at my office since our home ISP has a different rate cap for external traffic. This tends to be the same for other ISPs around here.. If you stay in their network you get better speeds.

[MoJo]

Honestly this is not about obtaining higher speed but rather dampening the effect of BT's abuse of bandwidth resources ... which may I add I applaud the adoption of uTP it does wonders in clearing congestion by a truly adaptive sensing mechanism. It is about doing the right thing and implementing it in the protocol level. ISP grade routers send traffic down the cheapest path, couldn't the BT protocol not do the same? It's about preferring the local traffic (ISP's in big cities tend to peer at internet exchanges ie. TORIX) over the costly global transits. Yes ISP's have given no incentive in rewarding the user (DPI's and Throttling) would make sharing across local loops slow but again it's more of cyclic blame game ... ISP's say they did these measures because of the abuse.

Lastly about BT choosing based on Latency I find that incredulous ... as far as P2P BT can handle high latency or small latency (When you get to VOD is when you want low latency). Other than that a file download won't see jitter based on latency, so latency shouldn't be the primary metric. BT as a protocol has a responsibility to be as proactive about congestion as it does encourage sharing of upstream bandwidth unlike any other P2P network and for cable co. Upstream is where it counts. I like uTP and how it has matured, and more should be done with respect to how to efficiently use systemic resources.

This idea if it holds the potential of reducing overall P2P out network traffic by even a percent it would have a significant dollar value for ISP's to stop coming after us the users and enforcing caps, installing more throttlers, etc...

Again our collective responsibility,

[Switeck]

Because ISPs jealously guard their peering arrangements as "company secrets", there's not much uTorrent can do no matter HOW well-meaning!

I've pointed out numerous times before that uTorrent can be made to run with far lower bandwidth "waste":

http://forum.utorrent.com/viewtopic.php?id=3474

http://forum.utorrent.com/viewtopic.php?id=58404

http://forum.utorrent.com/viewtopic.php?id=47523

However people are more likely to use very bad settings:

http://forum.utorrent.com/viewtopic.php?id=62085

Other posts on this message thread have already covered the other issues.

[MoJo]

Yet we still come back to this, it will make an impact even if you don't know Peering Contracts ... those aren't necessary when the simple dynamic of internal aggregate is cheaper than backbone transit. If one person in the city belonging to a certain ISP downloads a certain part it makes less sense that the n amount of people from the same ISP download from someone else across the pond when this peer is sharing that same part. I'm just not sure people understand the concept.

I say it as it is, this is not meant to speed up downloads but to increase BT's bandwdith usage efficiency. uTP is great, but it doesn't go far enough ... it only effects congestion, and scales back based on that. More can be done in making BT as a system a cost effective medium, this ensures that.

EDIT: Also Switeck those recommendations are quite standard, now to really push the envelop in making uTorrent reduce it's resource hog status this needs some serious thinking. Overall this would improve things on the ISP level and remove the stigma that BT currently has. ISP's don't need to play ball, they have money to invest in DPI and throttlers in that case we need to be proactive and show P2P is the least of there concern as Flash based video streaming becomes the norm.

[Switeck]

No, those recommendations are NOT "quite standard". They are force-fit into very limited constraints. They take the opposite approach (use fewer connections) to "slow speeds" than many guides suggest. (make more connections)

"this is not about obtaining higher speed but rather dampening the effect of BT's abuse of bandwidth resources"

If using one's line to its limits is "abuse" then clearly there's a huge user education process that's failed...and massive amounts of false advertising by ISPs. Some don't even mention the upload speed in their ads!

BitTorrent cannot use any more bandwidth resources than it's given. BitTorrent can already be set with limits below that. Running a few HTTP downloads or especially FTP downloads + uploads, they will attempt to consume all available bandwidth.

When tragedy of the commons occurs, it is only because ISPs have failed to continue upgrading their infrastructure to match the massive uptake in internet traffic over the last DECADE. They are still building their networks on the assumption that many if not most lines are idle. Provisioning is barely faster than peak dial-up speeds for DOCSIS 1.1 cable ISPs. Average contention rates are often 50+:1 on cable and even the "better" ADSL services are still 30+:1. This is even in the ISP's own internal networks for cable, the actual internet gateways offer even worse ratios!

Even a simple assumption that ips in the same #.x.y.z are more likely to be local, especially if only z differs, would be a start. But ignoring/avoiding international ips can result in dead torrent swarms as peers/seeds segregate into disconnected "islands".

[pan_torrent]

I don't care wether my peer is local or ISP outbound. I just connect to the 8 fastest peers.

Please check and comment on my method. I make utorrent continuously choose fastest (not exactly, I mean peers with u/l or d/l above my expectation) peers , and keep the peers until they are categorized as slow.

http://forum.utorrent.com/viewtopic.php?id=71756

[unxed]

And what about (optionaly) giving higher priority to peers got from tracker(s) added via bep-22?

Very simple algorythm:

1. If the peer is added manually, give it high priority.

2. If the peer has local subnet IP (192.168.*.*, etc), give it high priority.

3. If the peer is got from local tracker which is discovered via bep-22, give it high priority.

4. Otherwise, give the peer normal priority.

So ISP will be able to decide to have more local traffic or more outbound one by enabling or disabling local tracker or bep-22 support on it's DNS server.

No need for client to know ISP's peering policy, no need for subnet discovery, etc.

UPD: AFAIK, smarter algorythm is to give priority to peers got from any tracker which is in the same subnet with client.

[hackroute]

+ unxed

Very simple algorythm:

1. If the peer is added manually, give it high priority.

2. If the peer has local subnet IP (192.168.*.*, etc), give it high priority.

3. If the peer is got from local tracker which is discovered via bep-22, give it high priority.

+4. If the peer is in localpeer-list "localip.dat" (like ipfilter.dat), give it high priority and don`t disconnect (if it give a slow speed).

+5. if the peer is in my country, give the peer above-normal priority.

6. Otherwise, give the peer normal priority.

[GTHK]

If I get .2 kB/s from a peer and plenty of others exist then I want the bastard to die.