Insufficient Disk Space: NTFS & Files <4GB

[xlarbarx]

Windows XP Pro SP2

Using 1.8.11705; installed over 1.7.5

Getting Insufficient Disk Space errors with downloads that are less than 4 GB in size on a NTFS formatted drive, but there is more than 400 MB free space on the drive.

Simply restarting the torrents lets them continue downloading until the next time the errors occur. I have not been able to determine how long the downloads run until the errors occur, but this has happened several times.

I have several torrents that have been loaded but not started. If these torrents were to be queued/started, disk space would then be insufficient. However, there is enough disk space for the torrents that are currently queued/downloading.

Any suggestions? Possibly a bug?

[thelittlefire]

... 400 MB of space, in addition to the files already allocated. Remember unless you specify sparse_files in the Advanced screen of Preferences all the space is taken up at the beginning.

[xlarbarx]

Sorry, you lost me.

Diskio.sparse_files is currently set to true. Not sure if you're saying that I should have that set to false?

From what I understand, pre-allocating the disk space (in the general settings tab) should allocate the disk space when the torrent is started/queued, not when the torrent is loaded without starting it.

However, I haven't changed these settings from when I was using 1.7.5, and I was doing the same thing (loading but not starting torrents when I didn't have enough disk space) without any trouble.

[thelittlefire]

Pre-allocate with sparse files enabled doesn't make sense to me no matter how many times I read how they CAN be used together.

Sparse files are enabled by DEFAULT in 1.8, where it isn't in 1.7. Double check the problems don't exist in 1.7.7 and then use something like Process Monitor from http://sysinternals.com to verify some difference in the file calls between the two versions :/

[xlarbarx]

So basically what you're saying is for me to check pre-allocating of files, but set diskio.sparse_files to false.

I'll give that a try and post again if the same problem occurs again.

Also, as of this post I have gone from XP SP2 to SP3.

**UPDATE**

Ok, about 20 minutes after setting diskio.sparse_files to false, my low HD space warning came up in the system tray. Opening up Windows Explorer and refreshing, I could see that hard drive space was rapidly disappearing.

I changed the diskio.sparse_files setting back to true, closed uTorrent, waited and then restarted uTorrent.

Insufficient disk space errors were replaced by "Invalid Download State, Try Resuming."

Will try restarting the computer to see what happens.

**UPDATE 2**

After restarting, Windows Explorer shows only 320 KB (NOT MB) left on hard disk, whereas previously there was more than 200 MB. No new torrents were started or added. Where did all of the HD space suddenly disappear to, if the files were all pre-allocated??

Anyway, I've had enough fun with the RC. Will be installing the latest stable version.

[thelittlefire]

Once files are created, they cannot be made sparse. Similarly, sparse files cannot be made UN-sparse after creation unless you COPY the data. Invalid download state is interesting if uTorrent closed correctly. I don't see a need to preallocate, so try with that off, won't you? Leave sparse files on though.. Vista has problems with it, but XP is perfectly fine with it enabled.

[Firon]

You just don't have enough space on your computer. Really, this isn't a bug or anything. Sparse files just makes it take a bit longer to actually eat the space it's supposed to.

[xlarbarx]

Quote from Firon (an administrator on this forum) from http://forum.utorrent.com/viewtopic.php?id=3961 (Post #3):

"Pre-allocate (in µt) only does one thing: allocate all the files when you start the torrent to make sure you have enough disk space."

I like to pre-allocate disk space so that even if a torrent doesn't start downloading immediately, I will know that a torrent will use a certain amount of space on the drive when it actually begins downloading and can delay the start/queue of any torrents I may add later (because I'll know that I don't have enough free disk space). I mean, seriously, isn't that why a user would want to pre-allocate anyway?

The main point I'm trying to make here is that pre-allocating is working for files that I actually start when I open the torrent, but that something is causing ADDITIONAL hard disk space to disappear over time. Why?

[thelittlefire]

Check your volumes. See where your space is going. Compare it to the space you THINK it should be using. I recently came across http://windirstat.info/ which shows you graphically by type and location where your hard disk space is going.

[DreadWingKnight]

Pre-allocate on + sparse on = sparse on.

That combination causes the file names to exist, them to have a size in the properties of their actual size immediately, but their size on disk is the amount downloaded.

[Firon]

Sparse files must be disabled to use pre-allocate. Otherwise, you end up with this behavior where it looks like they're all allocated, but they're actually sparse and not really allocated. In 1.8, pre-allocate automatically overrides sparse, but not 1.7.7.

However, any file that was ALREADY sparse will stay that way unless you make a copy, or delete it.

So in effect, pre-allocate is not doing anything in your situation.

[GTHK]

Once files are created, they cannot be made sparse

Can be done using fsutil and NTFS file compression.

xlarbarx, have you tried NTFS file compression? It'll take a while if you have a lot of data but depending on the files and the structure it can gain a bit. Example, I had a video file that for some reason, had 5 MB worth of zeros on it. A small gain compared to the file being 180 MB but still. Also, Windows Explorer still counts sparse files allocated size against your free space unless you use compression. < /me Forgets what bug I was trying to remember.

[xlarbarx]

"Sparse files must be disabled to use pre-allocate. Otherwise, you end up with this behavior where it looks like they're all allocated, but they're actually sparse and not really allocated. In 1.8, pre-allocate automatically overrides sparse, but not 1.7.7.

However, any file that was ALREADY sparse will stay that way unless you make a copy, or delete it.

So in effect, pre-allocate is not doing anything in your situation."

There's the answer I was looking for. Thanks. Maybe clarify this for the next stable release?

[thelittlefire]

Have you read the manual? http://utorrent.com/download.php Ultima spent much time including useful information and troubleshooting information (I don't recall seeing THIS particular situation though) while still keeping it useful.

[Firon]

Sparse files aren't counted against free space. Only what they actually use is.

There's the answer I was looking for. Thanks. Maybe clarify this for the next stable release?

It is clarified already. Pre-allocate now overrides sparse, as it should.

[Ultima]

diskio.sparse_files: ...

- This option cannot be used in conjunction with pre-allocate all files. If both options are enabled simultaneously, pre-allocation will take precedence.

Very much in the manual.

[xlarbarx]

"diskio.sparse_files: ...

- This option cannot be used in conjunction with pre-allocate all files. If both options are enabled simultaneously, pre-allocation will take precedence."

If that's the case, then when my torrents which were loaded when I was still using 1.7.5 (where pre-allocation was enabled and sparse_files=false), then it wouldn't make sense that after I installed 1.8.11705 (pre-allocation enabled and sparse_files=true by default) I would be getting those insufficient disk space errors. Since pre-allocation was enabled when I was using both versions of uTorrent, sparse_files shouldn't have had an effect even if it was enabled. Yet I was continuing to lose disk space over time IN ADDITION to the disk space that was pre-allocated.

Tell me if this sounds like something that should be happening:

1) Version 1.7.5 (pre-allocation enabled; sparse_files=false which is default) Torrent loaded & started; 400 MB free disk space; 97% complete

2) Installed Version 1.8.11705 (no settings changed by me, but pre-allocation enabled and sparse_files=true by default)

3) New torrents loaded (but not started! Disk space should NOT be allocated as the torrents are not active). Still 400 MB free disk space.

4) Files which were already being downloaded before the upgrade to 1.8.11705 suddenly claim insufficient disk space. (This is where I'm confused. If the files had already had disk space pre-allocated, and pre-allocation over-rides sparse_files even if the value is set to true, then why would a torrent suddenly say that there's insufficient disk space?)

5) Restarted computer, ran scandisk, recovered 400 MB of space.

6) Restarted uTorrent. Errors now say "Invalid Download State: Try Resuming." Restarted torrent which is already at 97% complete.

7) Disk space begins rapidly disappearing. Again, why? The torrents that are active should already have had disk space allocated.

[Firon]

Something else is probably eating your space. Could be something as simple as an AV using a temp dir to unpack stuff. And the fact that scandisk "recovered" space makes that very likely.

In any case, make some space on your hard drive and stop whining. Windows and the filesystem need and use some HDD space.

[thelittlefire]

Did you try out the tool I mentioned... if it wasn't apparent, it allows you to see by LOCATION and FILETYPE

[xlarbarx]

"In any case, make some space on your hard drive and stop whining. Windows and the filesystem need and use some HDD space."

The downloads are on a separate physical drive.

Geez, why the harsh words? Whining? I thought this was troubleshooting of a release candidate that seems to be acting buggy.

Anyway, you don't have to start getting annoyed by this issue and start assuming that people are whining when they're just trying to take part in a troubleshooting process. I've rolled back to stable 1.7.7 and the errors haven't occured again.

[thelittlefire]

When you preallocate without sparse files you'll get those errors sooner rather than later, as the filesystem will (upon creation) use ALL the space of those files. So with multiple torrents started you need all the available space for both torrents at once whether or not they take 1 hour or 1 week to download.

[Firon]

If it was really a 1.8 bug, someone would've noticed it by now.

[xlarbarx]

"When you preallocate without sparse files you'll get those errors sooner rather than later, as the filesystem will (upon creation) use ALL the space of those files. So with multiple torrents started you need all the available space for both torrents at once whether or not they take 1 hour or 1 week to download."

I already understand that. My point is that torrents that were downloading already had the space allocated, and that space subtracted from the total free space on the drive when those torrents were started. There was still free space after those torrents were started, so if the reasoning is correct and pre-allocation is working correctly, then that free space shouldn't have been touched after the torrents began downloading. If the pre-allocation override is working correctly, then the whole discussion about sparse_files becomes pointless UNLESS this is a bug related to sparse_files (or sparse_files when used in conjunction with pre-allocation).

I'm going to make a guess at what's going on. When I was using 1.7.5, I had pre-allocation enabled and sparse_files was disabled (by default). After going to 1.8.11705, pre-allocation was still enabled but sparse_files was also enabled (again by default). Possibly, uTorrent somehow forgot the fact that my torrents had already had space pre-allocated, and was slowly allocating additional space because of sparse?

I'm not going to say that this HAS to be a bug; that's obviously something for the programmers to decide. I'll just say this: what I was doing worked for 1.7.5 and suddenly WASN'T working with 1.8.11705, and then went back to having no problems after going to 1.7.7.

Wouldn't anybody think that was kind of strange? I figured that if something unusual occurs after using a release candidate but works just fine using two different stable releases, it would only be helpful to post in your troubleshooting forum to identify what could POTENTIALLY be a bug.

Anyway, you guys can forget the whole issue. Like I mentioned above, I'm back to using a stable release and haven't had the same problems come up again.

[thelittlefire]

Well, I'm glad that's settled Sparse files are useful when you don't want lots of space taken up before you actually need it. Sparse files work by telling the OS "look we know it says this file is 3 GiB, but we only want you to write what we download now, capiche?" Most certainly all NEW files created while on 1.8 would be sparse-enabled so to get your space back to what you're used to follow the tip about "fsutil" on the command line or (my method) copy the files and start uTorrent back up. When updating to 1.8 with the (silent) change to sparse files setting... it explains the weird error in the client regarding downloading state. It doesn't nor can you adequately explain where the extra space went.

So given the isolated nature of the incident without more logging data for ex, doing your test case over with ProcMon (process monitor from http://sysinternals.com comes to mind) comparing actualized space usage with what should be happening indeed, it's something for another person at another time.