ISP request - help us to remove P2P throttling for our users

I work in ISP. We do the traffic throttling, to survive in the market and be competitive in pricing.

And yes, we want to deliver better speeds to our customers.

I read the following article recently:

http://torrentfreak.com/utorrent-2-0-to-elimininate-the-need-for-isp-throttling-091031/

and was surprised that (at last) someone from P2P is paying attention for the network side costs. Users pay them all, in the end.

The task to develop better TCP/IP (uTP) is huge. I really appreciate that this was started to reduce network congestions (=increase user experiences). But.... it can take very long time until it will become stable. And in the end this effort may fail also, if users will not tolerate a little bit lower speeds of uTP and the option will be available for them to go back to TCP/IP.

Much more simple solution would allow me to switch off throttling for uTorrent clients with huge reduction of costs, including increase in speeds for everyone. ISP and uTorrent developers shall cooperate more and I hope this post will add to that.

*** Background ***

The biggest problem (and the reason) why P2P traffic is throttled, is the network abuse from the P2P programs side. If you are in Sweden and calling your neighbour you can expect low rate for the local call. If you call Argentina, price is high, because traffic must pass Atlantic and many borders. Each byte that crossed Atlantic may be many times more costly than the byte that is delivered to neighbour. Long distance traffic is more costly for users, its not a problem only for operators, as some users think, because users always pay all the costs in the end. Megabit of traffic costs thousands in long distance and for some special nearby destinations, and cost nearly zero in short distance and for some longer distance other destinations. P2P clients do not care about that at all, happily downloading that 4MB torrent file piece from Argentina even when the source is available and offered by user nearby. This is an ISP tragedy that is unfortunatelly currently is solved by installing expensive traffic shapers that try to detect and limit P2P on "long distance" traffic. As algorithms are not ideal, in many cases the end result is that all P2P speed is reduced to keep web browsing available at peak hours. P2P developers put their creativity, time and resources to fool traffic shapers but in the end this only makes things worse because all the traffic is limited if shaper is not able to detect P2P traffic. Instead, would be much better if that programming energy would be used to offload intenational backbones and create more faster local connections for the users.

*** Proposed solution ***

Whats needed is the connection priorities, based on IP ranges. The client must get information directly from ISP about what connections are fastest and with lowest latency.

If many sources are available and offer the required content, client must connect for download to first priority IP ranges, then to second priority IP ranges, and so on. Its simple to implement, compared to uTP. The proposed IP_PRIORITIES.DAT file layout can be as this:

84.55.64.0- 84.239.255.255, 1

84.240.64.0-85.205.255.255, 2

85.207.0.0- 85.232.127.255, 3

85.232.160.0- 85.255.47.255,3

and so on.

The meaning is:

1. First, connect to sources that are in 84.55.64.0 - 84.239.255.255 ip range. If none available, or not enough sources here, then connect to sources that are in 84.240.64.0 - 85.205.255.255 ip range, then connect to sources that are in any level 3 range, and so on. If there is not enough sources in all ranges, then connect to any source, as it is now. If speed must be dropped (disk saturated, etc.), first disconnect (or reduce traffic to the minimum, if connection must be kept) from sources that are not in list and then starting from highest ranges.

The format is very similar to that of IPFILTER.DAT.

The file will be prepared by ISP. The client must get it by, first, resolving its reverse DNS name out of its external IP address. Lets assume uTorrent will resolve its own reverse DNS as this:

static-84.55.64.11.lakecity.westregion.startelecom.com

so, the client must try to http connect to these DNS addresses to download IP_PRIORITIES.DAT file.

_ip_priorities.lakecity.westregion.startelecom.com

if DNS not resoved, then try in this order:

_ip_priorities.westregion.startelecom.com

_ip_priorities.startelecom.com

So, first http request must go to

http://_ip_priorities.lakecity.westregion.startelecom.com/IP_PRIORITIES.DAT

next to:

http://_ip_priorities.westregion.startelecom.com/IP_PRIORITIES.DAT

and so on until the last one:

http://_ip_priorities.startelecom.com/IP_PRIORITIES.DAT

Thats all whats needed. Big ISPs will have several different files by region, small ones will have just one file. If all these DNS names are not resoving, then client will behave as it behaves now, without priorities.

Priorites, if downloaded, must be utilized on best effort basic on all sources (DHT, tracker, PeerExchange sources).

*** Why this will work? ***

Only popular files are the problem for ISP. I do not care for rare files - they never create so huge traffic that is able to kill web browsing on network backbones. The congestion problems are created by MOST POPULAR files only, usually just released. "Most popular" means there are lots of sources for them, sometimes thousands, so the proposed priority mechanism will increase the uTorrent user speeds by allowing client to connect first to the peers that are offering content on nearby 1000Mbps or 100Mbps links to the homes instead of trying to connect to peers that are on slow wireless networks, like GSM user with 14kbps, or to peers, for example, in South America.

If this will be implemented, I will advice all hundreds of thousands of clients to use only the P2P clients that have it. That message can be delivered with monthly invoices, for example, in start-up/novice guides, etc. I will also work with local torrent communities so that they accept only clients that observe IP prorities. The information is completely open - any P2P program can use the file.

This WILL work, at least for ISPs that do care. And many will care. I have contacts (agreements, etc.) with hundreds of other ISPs and of course I will explain for them that they must start to use the IP Priorities feature to increase their end-user speeds without raising interconnect costs.

As an ISP I know what links (IP ranges) have best speed, lowest latencies, and usually they are the cheapest also, so I can expand them easily if they are saturated, and usually expansion on these links will cost less that P2P traffic shaping solution. I want to invest in faster pipes and not in P2P traffic shaping solutions but some directions are simply too expensive. I would like to provide this network information to P2P programs and expect that they must consider the information when there are choices. Me too - I want, as you, the P2P programmers, to deliver the maximum possible speed for my limited interconnect budget I have. The interconnect speed expansion budget is limited because my users cannot pay more for their Internet lines. IP priorities would allow to eliminate all throttling, as users will no longer try to connect to South America for the most popular files, when there are 1000 of seeds locally.

I analyzed ISP.BEP22 option but that will not work. Not only because it is off by default, but because majority of users are in closed torrent sites, at least in my country. Theoretically, the similar result could be achieved by properly programming torrent trackers but usually their owners have no resources to reprogram and also DHT does not use trackers, so the client that wants to download must care about from where to try to get it first. Only downloading client must utilize the IP priorities. Upload must be offered to anyone as it is now, so that no one is discriminated.

Most important thing is that IP_PRIORITIES.DAT must be ON by default. No ordinary user will have a chance to register DNS service address with such a name, because ISPs are in control of DNS servers, in most cases deliver DNS addresses to users by DHCP, and also the reverse DNS name is the one belonging to ISP.

For ISPs that do not care to provide info about their networks to P2P programs, or those rare ISPs that have the same costs for any direction, then uTP may help. IP_PRIORITIES.DAT and uTP can complement each other, but IP priorities are easier to implement compared to uTP, and could be enforced very fast, so we can switch off throttling fast (or reduce impact of it).

The proposed solution does not require any changes on networks (routers etc.). IT departments will usually be able to implement this, by adding one DNS record and providing one text file.

Best regards,

Algis

Already being researched...and not doing very well:

http://forum.utorrent.com/viewtopic.php?id=31519

Where can I find the description of isp. advanced settings in 3.0 client? None in help file.

isp.fqdn

isp.peer_policy_enable

isp.peer_policy_override

isp.peer_policy_url

isp.primary_dns

isp_secondary_dns

There is small hope again that an "IP cost table" functionality, described in initial post, will be implemented and this still so important P2P program will pay attention to the traffic costs.

Since the first post, trend changed and P2P traffic % is now reducing (Internet is slowly transforming to VideoNet by traffic volumes), but still P2P creates huge part of load on backbones. Video services like YouTube are implemented by professionals and they do not create loads on international backbones, traffic is localized (otherwise YouTube alone would have killed the backbone). You never watch popular YouTube clips from other continent (even country and ISP, in many cases). I was part of YouTube traffic localization project. CDNs (content delivery networks) are built so that cost is lowest possible per byte delivered.

uTorrent itself is a big distributed content delivery system. Unfortunatelly not so much yet is done to reduce costs of torrent traffic (localize it). I notice here incorrect thinking that one fast far-away peer is better than 20 local peers given total speed is the same. But COST to transfer the file is not the same.

Good news is that BEP22 is on by default from version 3.0, but majority of traffic is still from private torrents, at least in my area.

By the way, my ISP removed any traffic shapers as international traffic costs went down. So majority of users now get 80Mbps and best plan users get 300Mbps for any destinations, no limits. But if traffic would be more localized, we could provide 500Mbps or 1Gbps speeds for the same monthly cost.

Please explain to ISPs what these ISP options mean. Can we use them to increase total network throughputs and rise customer line speeds, by localizing more traffic.

Example of realisation isp.peer_policy_url http://ls.orionet.ru/btpolicy.xml

Russian discussion http://forum.nag.ru/forum/index.php?showtopic=70838

<btpolicy version="1.0">
<revision>
2
</revision>
<oper>
Orionet
</oper>
<message>
Test rules
</message>
<ttl>
5
</ttl>
<!-- Private networks -->
<iprange start="10.0.0.0" end="10.255.255.255" weight="9"/>
<iprange start="172.16.0.0" end="172.31.255.255" weight="9"/>
<iprange start="192.168.0.0" end="192.168.255.255" weight="9"/>
<!-- Infanet-orionet -->
<netmask mask="94.232.56.0/255.255.240.0" weight="7"/>
<!-- Akvilon -->
<netmask mask="91.216.72.0/255.255.255.0" weight="7"/>
<netmask mask="91.229.208.0/255.255.254.0" weight="7"/>
<!-- Beeline -->
<netmask mask="77.240.144.0/255.255.240.0" weight="7"/>
<!-- Etherway -->
<netmask mask="91.197.172.0/255.255.252.0" weight="7"/>
<netmask mask="109.248.128.0/255.255.128.0" weight="7"/>
<!-- Info-link -->
<netmask mask="85.234.0.0/255.255.224.0" weight="7"/>
<!-- Nktv -->
<netmask mask="46.174.80.0/255.255.248.0" weight="7"/>
<netmask mask="178.213.208.0/255.255.248.0" weight="7"/>
<netmask mask="213.5.216.0/255.255.248.0" weight="7"/>
<netmask mask="213.108.144.0/255.255.248.0" weight="7"/>
<!-- Rostelekom -->
<netmask mask="79.133.128.0/255.255.248.0" weight="7"/>
<netmask mask="79.133.136.0/255.255.248.0" weight="7"/>
<netmask mask="79.133.144.0/255.255.248.0" weight="7"/>
<netmask mask="79.133.152.0/255.255.248.0" weight="7"/>
<netmask mask="89.151.128.0/255.255.224.0" weight="7"/>
<netmask mask="89.151.160.0/255.255.224.0" weight="7"/>
<netmask mask="95.81.192.0/255.255.248.0" weight="7"/>
<netmask mask="95.81.200.0/255.255.248.0" weight="7"/>
<netmask mask="95.81.208.0/255.255.248.0" weight="7"/>
<netmask mask="95.81.216.0/255.255.248.0" weight="7"/>
<netmask mask="95.81.224.0/255.255.240.0" weight="7"/>
<netmask mask="95.81.224.0/255.255.248.0" weight="7"/>
<netmask mask="95.81.232.0/255.255.248.0" weight="7"/>
<netmask mask="95.81.240.0/255.255.240.0" weight="7"/>
<!-- Chebnet -->
<netmask mask="31.148.0.0/255.255.240.0" weight="7"/>
<netmask mask="31.148.16.0/255.255.248.0" weight="7"/>
<netmask mask="31.148.28.0/255.255.252.0" weight="7"/>
<netmask mask="31.148.32.0/255.255.224.0" weight="7"/>
<netmask mask="31.148.64.0/255.255.224.0" weight="7"/>
<netmask mask="31.148.96.0/255.255.240.0" weight="7"/>
<netmask mask="46.187.0.0/255.255.192.0" weight="7"/>
<netmask mask="46.187.64.0/255.255.192.0" weight="7"/>
<netmask mask="95.141.224.0/255.255.240.0" weight="7"/>
<netmask mask="95.141.225.0/255.255.255.0" weight="7"/>
<netmask mask="95.141.226.0/255.255.254.0" weight="7"/>
<netmask mask="95.141.228.0/255.255.254.0" weight="7"/>
<netmask mask="95.141.229.0/255.255.255.0" weight="7"/>
<netmask mask="95.141.230.0/255.255.254.0" weight="7"/>
<netmask mask="95.141.232.0/255.255.255.0" weight="7"/>
<netmask mask="95.141.233.0/255.255.255.0" weight="7"/>
<netmask mask="95.141.234.0/255.255.255.0" weight="7"/>
<netmask mask="95.141.235.0/255.255.255.0" weight="7"/>
<netmask mask="95.141.236.0/255.255.255.0" weight="7"/>
<netmask mask="95.141.237.0/255.255.255.0" weight="7"/>
<netmask mask="95.141.238.0/255.255.255.0" weight="7"/>
<netmask mask="95.141.239.0/255.255.255.0" weight="7"/>
<netmask mask="213.88.0.0/255.255.128.0" weight="7"/>
<netmask mask="213.88.2.0/255.255.254.0" weight="7"/>
<netmask mask="213.88.2.0/255.255.255.0" weight="7"/>
<netmask mask="213.88.3.0/255.255.255.0" weight="7"/>
<netmask mask="213.88.4.0/255.255.254.0" weight="7"/>
<netmask mask="213.88.4.0/255.255.255.0" weight="7"/>
<netmask mask="213.88.5.0/255.255.255.0" weight="7"/>
<netmask mask="213.88.6.0/255.255.254.0" weight="7"/>
<netmask mask="213.88.6.0/255.255.255.0" weight="7"/>
<netmask mask="213.88.7.0/255.255.255.0" weight="7"/>
<netmask mask="213.88.10.0/255.255.255.0" weight="7"/>
<netmask mask="213.88.16.0/255.255.240.0" weight="7"/>
<netmask mask="213.88.32.0/255.255.240.0" weight="7"/>
<netmask mask="213.88.48.0/255.255.248.0" weight="7"/>
<netmask mask="213.88.56.0/255.255.248.0" weight="7"/>
<netmask mask="213.88.64.0/255.255.192.0" weight="7"/>
</btpolicy>

Thank you hjhjhj, Its totally amazing that this was implemented! This feature must be put into mass use by as much ISP and as soon as possible, so we raise Internet speeds ASAP.

I hope the uTorrent people who have the specifications for these settings will pay attention soon and post the official instructions about all the ISP. options, including information how to supply the policy file to everyone with uTorrent version 3.

http://forum.utorrent.com/viewtopic.php?pid=621170#p621170

&

http://forum.utorrent.com/viewtopic.php?pid=621212#p621212

why not copy-paste (with a little modification) code from bep22 dns query for filling isp.peer_policy_url ?

like _bittorrent-policy._tcp.something.com

dns answer

priority = 5

weight = 0

port = 80

svr hostname = host1

and client fill isp.peer_policy_url with http://hosts1:80/btpolicy.xml

it would be better that inform all users about it. if you have > 10k users this is a big problem.

up. please do this.

Nuuuuuuuuuu not xml X<