Smaller torrents download FAST; larger ones SLOW; upload unaffected

[birdbeast]

Greetings

I have been using uTorrent for the past year or so, after having tried 2-3 other clients. I found that uTorrent not only provided the best performance, but I'm a huge fan of small-footprint software. Kudos.

Anyway, I'm an IT professional with over 11 years of experience (not trying to sound arrogant or snotty, just indicating that folks can speak with/to me in technical terms). My BitTorrent machine is a Pentium 4 3.0GHz, 512k L2 cache, 800MHz FSB (yeah, the OLD Pentium 4s), 1GB RAM, BroadCom 440x 100BASE-TX wired ethernet (not using wireless until I get this figured out, if possible), etc.

My ISP is Earthlink DSL providing an advertised 3Mbps down and 768kbps up (unfortunately the highest amount of bandwidth available at my address -- literally across the street is serviced by RCN offering 20Mbps down and up to 8Mbps up, but you take what you can get). The dslreports speed tests report ~2.94Mbps down and ~640kbps up. In practice, downloading via HTTP from fast servers, I have seen up to 440KB (kiloBYTES)/sec; approx. 3.52mbps) down, and maybe up to 70KB/sec (560kbps) up.

My router is something of a cobbled-together beast; it's an old machine (Pentium II 300MHz, 160MB RAM) on which is running a custom built Linux (kernel 2.4.x) on CD. It boots directly from the CD-ROM, loads the OS and filesystem into a RAMdisk, and configures itself from a write-protected floppy -- no HDD required (though I have one installed in the machine which I mount from time to time to create a backup image of the floppy). The router uses iptables for port forwarding; I have forwarded port 9344 to my BitTorrent machine and do see the green connected icon at all times when downloading a torrent. I have a series of cronjob scripts that run once every five seconds to detect if my IP has for some reason changed, in which case the firewall script containing all of the port forwardings is re-executed for the new public IP address. I am doing nothing fancy with the router -- a bunch of unique forwarded ports (no ports are "re-used" including 9344), no changes to the routing table, etc.

My BitTorrent settings are currently set at the settings at which the Speed Guide configures them for a Connection Type of "xx/640k" -- that is, an upload limit of 60kB/s, 4 upload slots, 100 connections per torrent, 375 global connections, 4 max active torrents, and 3 max active downloads. As I said, port 9344 is the port I use, and the "Test if port is forwarded..." button does return a green "OK" confirming that the port is forwarded properly from my public IP. Enable Encryption is checked.

That said, I have tried using quite a range of settings -- upload limits of 25k/sec - 60k/sec, global connections of 100 - 800, per-torrent connections of 50-400, etc.

Now to my actual problem.

I generally do not attempt to download a torrent unless there are at least 5 seeds and 50-60 peers. When the torrent has, say, between 5-20 seeds and 50-150 peers, uTorrent is maxing out my connection within 15 minutes of starting the download, with very little (if any) fluctuation. No problems there -- happy as a clam.

However, for very popular torrents, when there are hundreds of seeds and thousands of peers, things slow down dramatically. For example, I have been downloading a current torrent of size 7.46GB that has 156 seeds and 1306 peers for almost exactly 12 hours now. My download rate has averaged between 40 - 100k/sec. There have been "peaks" of up to 280k/sec, but they last for 60 seconds at best and they have only occurred 5-7 times since the torrent was started. Currently, I am showing a download speed of 48k/sec, and an upload speed of 53k/sec, with 8 / 156 seeds connected and 74 / 1298 peers connected.

Upload speed is generally not affected -- for both torrent "sizes," it remains fairly pegged at the maximum configured in the settings. It is only the download speed that fluctuates, and quite significantly. I have observed this phenomenon now over the course of three separate ISPs -- first Verizon DSL, then Service Electric Cable/ProLog, then EarthLink DSL. However, during my time with ProLog, I was using a different router (some LinkSys or NetGear piece of trash); I did have a port forwarded during that time as well, though.

CPU utilization of uTorrent on the BitTorrent machine ranges from 0 - 3%; overall CPU utilization (all running processes) ranges from 0-7% according to Windows Task Manager. Overall CPU utilization (all processes running) on the router machine is 1-5%. Currently it has been up 12 days, 9 hours, 33 minutes, with an average CPU load of 0.12% (that is 12/100 of 1%, not 12%).

On the BitTorrent machine, there is currently 675MB of physical RAM free; on the router, there is currently 75MB of physical RAM free.

This problem has me pretty much stumped, mostly because everything works flawlessly as long as the torrent isn't too popular. This tells me that I most likely don't have a settings problem (although I'm not totally sure, I'd really like to understand the uTorrent algorithm more in-depth), and seems at first glance to most likely be a resources problem. However, the absolutely tiny loads on the BitTorrent machine and the router seem to defeat that likelihood.

Any help is appreciated. Specifically, is there any more technical reading available on the uTorrent algorithm with regard to peer selection/preference? When I examine the Peers tab during the aforementioned problem, I see that the absolute majority of peers are not (and have not for some time) actively downloading from or uploading to me at all. I do not pretend to know anything deeply technical about the general BitTorrent algorithm/strategy or more specifically uTorrent's implementation of it, but it would seem to me that the peer list would be pruned periodically to "dump" inactive peers and connect to new ones until the client is connected to the most productive "n" peers in the swarm, where "n" is the torrent connection limit set by the user. Is there some property of the algorithm or model that prevents this kind of intelligent peer selection? Even without the answer to this specific question, I would find any technical read on the uTorrent or BitTorrent algorithm edifying, and would appreciate any links thereto.

Finally, I have not touched any of the advanced settings. Is there some sort of guide that explains what each one does? Do any of them jump out as potential culprits of this particular situation?

I do not and have not seen any error messages during my operation of uTorrent in the past year.

Thanks for any information.

-BB

[Switeck]

Sounds like a subtle overload with "huge" torrent swarms.

What half open connection limit are you using?

µTorrent tries to cycle through them all, and maybe it's getting lots of improperly disconnected ips...which add up and then cripple your download/upload speeds.

(Your upload speed is crippled too, right?)

This will probably help too:

http://forum.utorrent.com/viewtopic.php?id=15992

[birdbeast]

Right; as your forum sticky advised, I read that entire post and followed all of its suggestions before posting on my own.

As my post says, my upload speed does not suffer at all during the periods of download speed problems.

Can you answer any of my other questions regarding technical material on the BitTorrent algorithm?

As I mentioned in my post, I don't know how the advanced settings affect the application, nor am I aware how tweaking the TCPIP.SYS file affects the application (while the FAQ provided links to actually perform the tweaking, it did not explain what the tweak actually does; furthermore, it claimed that Windows XP SP2 users do not need it, which is what I am running).

Having never changed the half open connection limit (I assume you are referring to the advanced setting "net.max_halfopen," it is currently set to 8. Should this be changed? As my original post asked, is there any documentation available on the advanced settings? And are there any other settings that I might change to experiment with solving this problem?

Thanks for any help

-BB

[kurahashi]

furthermore, it claimed that Windows XP SP2 users do not need it, which is what I am running

Exactly the opposite - it's the WinXP SP2 users who do need it.

Too low half open conneciton limits can obstruct the proper 'flow' of peers, making in result those bad (useless) peers still connected to you while not being able to connect the others, possibly better.

I'd suggest trying to patch your system to 50 half-open limit, while trying to increase half-open limit in utorrent to 20 and see the results.

Nevertheless, you still have to remember that bigger torrents will always have worse speeds. It is because of lower seeders ratio of such, people are tending to seed such torrents for shorter time because of the greater hdd space it consumes and/or the natural temptation (and possibility) to archive this on CD/DVD.

[Switeck]

I would expect an improvement with reduction of net.max_halfopen from the default of 8 to 1-4.

I doubt Windows XP is the bottleneck.

Something else (probably software firewall/antivirus/antispyware software) is probably experiencing an overload or bug due to µTorrent's network traffic.

[birdbeast]

As I mentioned in my original post, I do not run any software firewalls. Nor do I run any awful resident applications like antivirus or "antispyware software." If I suspect a virus or spyware issue, I run a single system scan with a small application like Ad-ware. Nothing stays resident in memory on any of my machines.

I will try the two suggestions, although one suggests more than quadrupling a configuration value and the other suggests halving it... Oh well. I guess all I can do is experiment.

Thanks anyway

BB

[Switeck]

If it's a stability issue, I seriously doubt quadrupling half open max value is going to make it MORE stable.

kurahashi is just assuming you're getting slow speeds due to low connection rate.

But if you're not firewalled in µTorrent then low half open rate shouldn't be a problem on anything other than nearly dead torrents that are reporting lots of bogus peers/seeds with a tiny few good ones mixed in.

[drbits]

I have been having similar problems myself. I seem to have been able to more than tripple my download speed for a single large file.

I made a combination of two changes:

1) Changed the system half-open connections to 100 and set net.max_halfopen to 50.

2) I opened the "Peers" tab and set it to sort by "Down Speed" (maximum at the top).

In my case, I am only downloading one Torrent (32GB) with seeds 4(20) and peers 60(260). I am stuck with a low speed line (768/384 kb/s) with DSL reports showing about (600/320 kb/s).

My upload rate is fairly constant (I have many files to upload, limit set at 24kB/s to allow for other use of the machine). My download rate varies from 15kB/s to around 50kB/s (averaging around 30).

Both changes had a significant effect, but change 2 was more effective. This is probably a bug in uTorrent. I am guessing it is bug in the "roaming pointers" that poll peers.

By the way, I have a similar computer, except that it is only 2GHz and a slower FSB. uTorrent is using about 2% of my CPU time (total use about 10%). I am running a TwoWire router and software AV and anti-spyware, they have little effect on these transfers (they mostly work on file open and close, not during transfer).