Speed up file allocation

[chupakabra]

When allocating large files uTorrent has to fill them with zeroes. That creates significant disk load and affects overall system performance. Windows XP provides a way to allocate files of any size almost instantly: SetFileValidData() function. It won't work without SE_MANAGE_VOLUME_NAME privilege (admin rights I guess) and with compressed files but I think it won't hurt anything in case of failure. Here is a link to small program that demonstrates this concept: http://depositfiles.com/files/8072561

[Firon]

It is for these reasons that SetFileValidData is not recommended for general purpose use, in addition to performance considerations, as discussed below.

You can use the SetFileValidData function to create large files in very specific circumstances so that the performance of subsequent file I/O can be better than other methods. Specifically, if the extended portion of the file is large and will be written to randomly, such as in a database type of application, the time it takes to extend and write to the file will be faster than using SetEndOfFile and writing randomly. In most other situations, there is usually no performance gain to using SetFileValidData, and sometimes there can be a performance penalty.

Plus the requiring of admin rights just will give more headaches and require special casing. Not worth it. Turn on sparse files if you really, really don't want zeroed out files.

[chupakabra]

there is usually no performance gain to using SetFileValidData

Just run the attached program with and without SetFileValidData() and you will see significant difference in performance. uTorrent allocates files the same way.

Plus the requiring of admin rights just will give more headaches and require special casing.

What headache? Run attached code without admin rights and you will see that it still allocates a file. In this case SetFileValidData() just silently fails without breaking anything. Subsequent writes will set file valid data but with performance penalty caused by filling file with zeroes.

Turn on sparse files if you really, really don't want zeroed out files.

Not a solution. Enabling sparse files causes severe fragmentation and slows down file access significantly.

[Greg Hazel]

There are some security considerations here. Since uTorrent shares read access, allocating a file without zeroing it means it could expose data on disk to a non-admin user.

That said, as an option and with correct user awareness, I think it's a good idea. Thanks for the sample code!

I wish Windows sparse files worked this same way, but returned zeros when you read un-written-to sections of the file.

[Greg Hazel]

This is implemented in 1.8.1 build 12320 as diskio.no_zero

http://forum.utorrent.com/viewtopic.php?id=44704

Give it a spin and let me know that it works for you.

[Firon]

Figured it wasn't a bad idea to try if it just fails silently and still allocates a file normally, as you mentioned.

[chupakabra]

Thank you for implementing!

During testing I noticed:

1. Need to restart uTorrent for diskio.no_zero to take effect.

2. diskio.no_zero only works when 'Pre-allocate all files' is enabled.

3. Not working with magnet links ('Pre-allocate all files' seems not to work also in this case).

[Greg Hazel]

Restarting is not surprising - it is only applied when the file is opened.

It is surprising that you only see it function with pre-allocate, since it is applied when ever a file is opened for writing. All pre-allocate does is open the file for writing right away. So, what behaviour do you observe with pre-allocate off that leads you to believe it's not working?

[chupakabra]

Restarting is not surprising - it is only applied when the file is opened.

This is how I tested step by step:

1. Start uTorrent build 12320 for the first time and enable diskio.no_zero.

2. Start downloading new large torrent - diskio.no_zero not working (slow file allocation).

3. Delete .torrent+data and restart client.

4. Start downloading the same torrent again - diskio.no_zero works.

So, what behaviour do you observe with pre-allocate off that leads you to believe it's not working?

As usual: high disk activity, 'Disk overload 100%', download/upload speed drop until file is allocated.

[Greg Hazel]

Ah, I see the bug. Fixed, thanks!

[Psi]

Thanks for implementing it! That's the only thing in utorrent that disturbed me.