Disk overload when DL torrents at 17 MB/s

[Slangaren]

Hello.

I have run uTorrent for a while and I ran into "disk overload" problems also at 6 MB/s. But then I disabled "disk cashe" in Windows and everything was fine.

Now I changed my FW/Router in the weekend to one (D-Link DIR-655) that can handle 266 Mbit/s in throughput (WAN-LAN). So I got the "Disk overload" problem back.

I have 1 GB of memory so I allocated 700 MB to uTorrent.

I have no FW or antivirus installed.

Only uTorrent, W2003 server, WinRAR and eMule.

It starts fine when I start download with increasing DL speed but when it reaches about 16-17 MByte/s I get 100% disk overload and the DL speed goes down to 3 MB/s.

I checked reading from the disk and can do so across the network at 250 Mbit/s, s the disk should be able to handle 130-140 Mbit/s in DL..

Any ideas?

(Can't attach the diskcashe screendump)

Everything in the Diskcashe page is chosen except the "reduce memory..." and "write out finished..."

And 700 MB is allocated..

[DreadWingKnight]

bittorrent downloads aren't sequential, so the disk accesses probably aren't sequential either. Because of this, your disk throughput is reduced to a percentage of the drive's total capabilities.

This disk overload happens when the hard drive's speed, the windows cache and the uTorrent cache can't keep up with the speeds you're sending at them. It's a case of the slowest man dragging everyone back. The non-linear disk accessing is the problem.

[Slangaren]

I thought that was one of the reasons why you assign a lotof memory to uTorrent.

With 700 MB it should be possible to even out the writes..

[DreadWingKnight]

Because of the nonlinear download method, 700mb isn't always enough.

[PeeBee]

I don't have a solution, but... what the hell kind of torrents do you find that will download that fast?

I have a 100 MBit FibreLAN line, a Netgear WNR854T gigabit capable router, and two PCs with wired gigabit LAN. According to speed tests I get roughly 90% efficiency out of the 100 MBit line, allowing me to download at a maximum of about 10.7 MB/s. I've been getting up to 9 MB/s on HTTP downloads. But the fastest I've ever seen in µTorrent is 1 MB/s. Usually it's 1-2 MB/s upstream and around 20-200 kB/s downstream with the occassional 500+ kB/s spike. In the preferences I have unlimited UL/DL speed, port forwarding works, I see a whole bunch of DHT nodes, everything's fine and dandy.

[Slangaren]

What do you mean by "nonlinear download"?

I hear what you are saying, but i get a very steady download which peaks at 17 MB/s.

As the disk can handle around 30 MB/s (250 Mbit/s), I can't see that this is the problem.

And 700 MB is about 1 min of download, and this should more than enough handle any "even out" of flucuating data stream.

[Slangaren]

I get the torrents from "Powerbits", they are generally very fast.

The router you are using can handle 92 Mbit/s in throughput according to:

http://www.smallnetbuilder.com/routerchart/charts/index.html?chart=119

But this is one filetransfer and not many sessions like in torrents.

Some people say that torrent packages are tougher for the routers FW than straight FTP. So you might not the 92 when using torrents..

[DreadWingKnight]

Take a look at the pieces tab while you're downloading.

The first column lists the piece number.

You'll see that the piece download order (while watching the blocks fill in) is not sequential.

[Slangaren]

Ok,

I get your point.

Have to look into that.

Is there a way for uTorrent to priorotize between blocks? Or any other solution.

I have read that some people are using uTorrent on Gbit connection.

How do they do?

What maximum speeds have people downloaded torrents at?

It must be more than me having this problem..

[Switeck]

And your problem may be caused by multiple things at once.

Your router and/or modem may be bursting really badly because it cannot sustain the speeds you told it to use.

Your hard drive/s may be having problems with the intermittent burst rates.

And your cpu may be under a heavy load due to an unknown trojan/virus (for all we know)...or even poorly written commercial software! (Most "productivity" and tweaker software is junk.)

Under Disk Cache settings, disable:

write out untouched blocks every 2 minutes

turn off read caching if the upload speed is slow

If what you're uploading + downloading is less than 700 MB, then your hard drive/s should not be overworked at all because it should all fit in the disk cache!

[cbw]

I have the same problem.

My connection is PPPoE Broadband and I have 100Mbit/s metropolitan speeds.

The thing is that if I set the write cache to about 700 MB, it gets full very fast because the speed boosts to 10 MB/sec and uTorrent only writes to RAM, doesent write data from RAM to HDD (maybe couple of MB).Only when the write cache is full, it starts to write to disk, I get Disk Overload 100% and DL drops near 0 until the cache empties. All along this process, there is no Upload (rather annoying because you get low ratios).

If the automatic write cache is enabled, the speeds are stable at 5-6 MB/sec. I am sure my harddrive can sustain more than 5-6MB/sec random writing speeds. I'll try to post a HDD benchmark to proove that. I must say that the NIC is an expensive 3Com and the CPU load is resonable.

Is there any chance that you do a mod for uTorrent working in HiSpeed networks?

Thank you,

Cbw, Ro

[Lord Alderaan]

If it is one torrent running at 17MB/s try limiting that torrent to 15mb/s to prevent it from causing an overload.

If you have more then one physical disk drive you can try downloading different torrents to different drives instead of letting them all download to the default download folder. By spreading out the writing (and reading) over multiple physical disks you can handle downloading at much higher bandwidths (until you meet the next bottleneck).

Please note a physical disk is not the same as a drive/partition (such as c: and d:). One physical disk could have multiple drives/partitions. See the Disk Manager (r-click on My Computer -=> Manage) to find out which physical drives contain which partitions.

[cbw]

I belive that Slangaren has the same problem as I do, so I'll try to explain. With hiSpeed networks, the cache gets filled very fast, not only because of the speed itself but because there is no writing on the disk before the cache gets filled (700 MB for example).

Let's do the math: if uTorrent writes directly to the cache (RAM), wich is very fast, the speed limit is that of the NIC, Router and so on, therefore I have 10 MBytes/s dl speed until the cache is full.

After the filling of the cache, there is no more dl and uTorrent starts to write to disk and it writes very slow. I don't think that my HDD is so slow so that it couldn't write at least 5 MB/sec of (already) cached data.

You can observe all this in Speed->Disk statistics tab. (green thick-disk, green thin-cache)

Using auto cache size, the speed will drop when it's overloaded so it'll stabilize at about 6 MB/s.

I would like to know if it's possible to DL at max speed all the time...

[Slangaren]

I have started to check some things..

When I kick in a new torrent, CPU load goes up to about 80% and the PF usage increases linearly until reach about 800 MB and then "Disk Overload" appears.

I also checked the processes tab in taskmanager and there is no other process than uTorrent that is using any memory or CPU (<2MB and <1%).

I used the settings above without any change.

When I use "Write out finished pieces imediatly" there is no difference. Same DL speed, same disk overload problem.

My computer as described above is a 2 yr old laptop, only with W2003 Server without FW. uTorrent, emule, WinRAR installed. It is only for DL torrents and emule. Nothing else. No antivirus installed, and I would say no viruses could be there. Very fresh install of Win.

If I let the torrent start with a limit on the DL in order to find all seeds / peers first I could see a maximum DL of 25 MB/s before overload when I put DL to unlimited.. What is the maximum that other people with fast connections has reached?

If I look under pieces my colours are blue on not green as described by cbW. Can someone comment on the colours. Are the blocks "completed" when written to cash or disk? What does "Mode" mean?

If you limit the DL speed you can also see that it doesn't start to write to disk until the cashe is full..

Thanks

[Lord Alderaan]

From the fact that its a laptop I presume it only has one HD. In that case I'd suggest capping the overall download at 15MB/s or even lower (in steps of a mb) if that doesn't solve your problem

Even having to write to only two spots at almost the same time will influence a HD's writing speed. So even with a large cache it will bottleneck on a single HD. As soon as your cache is full it has to write data to you drive. The advantage of the cache then is that hopefully it has some of the pieces available that are needed for uploading (and thus reduce reading directly from the drive) and that it might be able to group pieces together for sequential writing but it probably still has to move over your hard disk rapidly which means it will never reach the speed a single large sequential write (copy from network) would be able to reach and it can overload at a 'mere' 17MB/s.

[Slangaren]

True,

But as I said, I can write to the drive at 30MB/s when moving a file on the LAN.

I even get disk overload at a limited DL speed of 3 MB/s when the cashe is full, but then it writes to disk and the overload goas away for a short time until full again etc..

I am using "write out finished pieces imideatly", but it still fills cashe before writing..

[Slangaren]

I checked some more under the speed / disk statistics tab.

And it looks like my disk is writing on a average of 1 time per sec. The size is about 1 MB.. If I stop the DL when the cashe is full (700 MB) it takes a long time (several minutes) to write out the cache to disk. It continues with same spee as when downloading.

My friend has a similar setup with a new desktop pc, and he has the same problem.

No one has written here that they DL quicker with uTorrent (to disk)..

There must be quicker out there..

[Lord Alderaan]

I can't speak from experience. The maximum speed I've reached with µtorrent is 1MB/s and it was a single torrent. So I'm sure my internet connection was still the bottleneck. I don't know which speeds are reasonable but non-sequential writing SERIOUSLY degrades performance! A cache is merely a tool to improve efficiency but can only help so much. Setting it bigger and bigger has less and less effect.

The fact you CAN download at 17 MB/s while the cache isn't full yet doesn't say anything about the performance of your hard drive. Only when the cache is full do you get accurate data about how much your drive can handle. Although 2MB/s seems a bit on the low side it could really be posible its caused by non-sequential writing. It would explain this problem guys problem too:

http://forum.utorrent.com/viewtopic.php?id=21684

However as I said I didn't do any testing myself and I don't know anyone who had speeds of more then 2mb/s on a single drive. I cba to go and setup a tracker in my LAN just to test the maximum speeds of µtorrent on different hard drive setups. So I can't be sure if it really is the limit of your drive or if something else is causing the problem. If it is something else I have no clue as to what it could be.

[Switeck]

Your hard drives may be stuck in PIO transfer mode due to windows crashes or poor configuration.

Check the hard drive controller speeds (PIO or Ultra DMA modes) under Device Manager under System in Control Panel.

[cbw]

Nope....Ultra DMA 6 here.

[Slangaren]

Same here.. Ultra DMA 5..

But I have some good news..

My friend has one Barebone with e brand new motherboard which he is using only for downloads / server at home. This mashine has similar problems as mine. Fills cache but then maybe 5 MB/s writing to disk (me 3 MB/s but on a laptop).

He has a new HTPC with two 2.5" disks in Raid0 which he also tried on.

This one looked completely different. When starting to fill cache 200 out of 300 MB, it wrote to disk. This writing was at about double speed of the DL speed (writing to cache) at about 22 MB/s. And this is how I was expecting it to work.. So half the time it was writing to disk and half idle on the disk, but all time loading cache..

We need to figure out where the difference is, but at least we have forund one computer which is working fine..

Will get back..

[Lord Alderaan]

He had only two disk in the Raid0... hmmz that shouldn't explain a increase from 5MB/s (maintainable constant speed on a single disk) to 22MB/s (maintainable constant speed on a 2 disk raid0 setup). Maybe the lack of running application writing and reading from the disk could explain it. And the fact that a raid0 can maintain higher speeds does point into the direction of non-sequential writing. After all Raid0's are the mother of all performance setups sacrificing all security for more performance (no redundancy has to be written but one single drive failure results in the loss of ALL data on all the drives ).

[lelik11a]

I had just the same problem... But only from one tracker where i download big files ( from 15 to 40gig). uTorrent was just writing to cache and not to file, there fore disk overload... My connection is 100/10, new hdds with 16mb buffer memory...

The only solution what worked for me is to start download, pause it after like 200mb, wait until everything went from cache to file and then just resume... After this simple "operation" it works like a charm! I download 40gig torrents with constant speed of 10-10,2MBPS without any problems... I didnt try to download few torrents at the same time, only one... But i will try it later...

Try it, may be it will work for you too...

[Lord Alderaan]

Interesting! I found it hard to believe only 2MB/s to 5MB/s would be sustainable...

May I inquire to your hard drive setup?

It could be some bug in µtorrents cache management or something. Maybe the devs could look into it.

[lelik11a]

I have 2 hdds, no raid or anything...

This is hdd i have http://www.samsung.com/Products/HardDiskDrive/SpinPointTSeries/HardDiskDrive_SpinPointTSeries_HD501LJ.asp