Downloads are always created as sparsefiles - still not fixed

[dermarcus]

Hi all,

I do not want the downloads to be allocated as sparse files. However i cannot configure uTorrent to do so. I am using uTorrent 1.8.1 beta build 11882 (newest version to date) with the following configuration settings:

append !ut to incomplete files TRUE

preallocate files TRUE

I can confirm that diskio.sparse_files is now enabled in advanced settings by default. Below is the list of advanced settings that i changed from default values. Although i cannot imagine them being relevant to the problem, i am including these to avoid being asked

gui.delete_to_trash FALSE

gui.update_rate 10000

queue.slow_dl_threshold 40000

queue.slow_ul_threshold 5000

I explicitly switched to build 11882 in order to have the sparsefile problem solved but sadly it did not help. At least not for me. I am using utorrent under a recent Wine. Could that be the problem? If yes, could this please be fixed?

Edit: I have made fresh installations on both Wine and Windows. The Windows installation seems to allocate files instantly as advertised (at least the windows explorer file dialog makes me believe so) however the Wine installation allocates files non-instantly. So the problem is in fact related to the way the file allocation is done (seeking, whatever) and how that is interpreted by Wine. Can someone from the developers please comment?

Edit: I have traced the system calls that utorrent does on beginning of download and have found the following sequence, which, in my understanding, does result in having the file created as sparse file. This sequence does nothing to prevent it from being created as sparsefile so Wine seems to be correct with its interpretation. Can someone from the developers please comment?

0018:Call KERNEL32.CreateFileW(0016d968 L"H:\\Star Trek Online Debut Trailer 720p (PC) - TGS\\star trek online debut trailer.wmv.!ut",c0000000,00000003,00000000,00000004,10000080,00000000) ret=0043312e

0018:Ret KERNEL32.CreateFileW() retval=00000194 ret=0043312e

0018:Call KERNEL32.GetFileSize(00000194,7e0a2928) ret=00412525

0018:Ret KERNEL32.GetFileSize() retval=00000000 ret=00412525

0018:Call KERNEL32.SetFilePointer(00000194,0281154d,7e0a2914,00000000) ret=0043320f

0018:Ret KERNEL32.SetFilePointer() retval=0281154d ret=0043320f

0018:Call KERNEL32.SetEndOfFile(00000194) ret=00412561

0018:Ret KERNEL32.SetEndOfFile() retval=00000001 ret=00412561

0018:Call KERNEL32.CloseHandle(00000194) ret=00433136

0018:Ret KERNEL32.CloseHandle() retval=00000001 ret=00433136

Best regards,

Marcus

[Greg Hazel]

That sequence of system calls does not result in a sparse file. DeviceIoControl(... FSCTL_SET_SPARSE ...) must explicitly be called to set the file as sparse.

I believe this is not true on ext2, where all that's required is that you do not write all of the bytes.

What filesystem is WINE running on top of?

[dermarcus]

It runs on an EXT3 filesystem and like you said this sequence creates sparsefiles then. I tried to 'solve' the issue by disabling sparsefiles outside of uTorrent but it did not find an acceptable solution to do so:

wine: does not have means to control it.

local filesystem: all local filesystems except FAT/VFAT support it and it cannot be switched of.

remote filesystem: samba server allows configuration of nosparse/strict allocation, however carrying a local disk through a loopback network filesystem layer is quite strange.

Do you have an idea?

[Greg Hazel]

Technically, if WINE is going to emulate NTFS in this way, it needs to write zeros to the file explicitly to keep EXT3 for making them sparse.

[dermarcus]

So utorrent does rely on the semantics of native windows NTFS which that creates sparse segments only when asked for explicitly? And Unix creates sparse segments automatically as soon as the filepointer is advanced behind the current end-of-file? Did i get that right?

If yes then i suggest adding the explicit zeroing to utorrent (sys.wine_hacks?) because it is significantly less work than changing Wine sematics. What is your opinion on that?

Edit: Running utorrent on Wine on NTFS is not going to help because ntfs-3g driver supports sparsefiles the same way as the other Unix filesystems.

[DreadWingKnight]

Sparsing when not explicitly told to would be considered a wine bug. It would need to be fixed at that level.

[dermarcus]

What exactly do you mean with: 'would need to be fixed at that level'?

a) It would be nice if Wine fixed it and we add this workaround in utorrent until it is fixed

It has to be fixed by Wine developers and you do not consider adding a workaround

Naively i would vote for adding a workaround (like many of the other workaround) because i consider the fix in Wine to be non-trivial. I do not know if Wine people do see the problem the same way as you. I will take a look into the Wine community to find out.

[Greg Hazel]

Hm, a work around is possible, but a little bit of work. How critical is this issue?

[dermarcus]

First of all thanks for your cooperation. I already submitted a bug report under http://bugs.winehq.org/show_bug.cgi?id=14914 but considering the length of the outstanding bugs list i do not expect a response anytime soon.

Personally i would appreciate a solution in maybe a week? Currently i have removed all content from my uTorrent machine in order to put it back online as soon as the fragmentation issue is solved. It has hit me so hard that some files would not even be read with 1MB/sec so i had to take action.

[Greg Hazel]

Unfortunately I will be out for two weeks starting tomorrow. You could manually create/extend the files. A simple python script would work:

import os
import sys
fname = sys.argv[1]
length = int(sys.argv[2])
f = open(fname, 'wb+')
b = chr(0) * 65536
while length > 0:
    f.write(b[:length])
    length -= len(
f.close()

[dermarcus]

Thanks, i can live with that timeframe as well. In the meantime i will check whether i can automate running your script somehow but i suspect it might collide with utorrent trying to write data as well. Not collide in the sense of data corruption but more in the sense that utorrent might pause the particular torrent because it could not get 'exclusive' access to the files. Time will tell...

Best regards,

Marcus

edit:

@alus

Has the proposed workaround (for Wine) already been implemented or considered worth implementing?

[arfyness]

Sparsing when not explicitly told to would be considered a wine bug. It would need to be fixed at that level.

I certainly must agree, but WineHQ apparently does not consider this to be a problem with Wine!

Certainly Windows' native behavior in this case would be to create non-sparse files...

See also: http://bugs.winehq.org/show_bug.cgi?id=14914

Even though this thread is more than a year old, it is still apparent on my Linux system, that running uTorrent in Wine causes (especially large) files downloaded to an ext2 filesystem to be *very* fragmented. Like Marcus, I have found that It doesn't seem possible to disable sparse files except at the level of the linux kernel (ext2/3 kernel module) which is as good as no solution at all.

If there is already a uTorrent work-around for the unexpected Wine behavior, I am simply ignorant of the correct settings in uTorrent to achieve this, please advise. ;o)

Current settings:

General: Pre-allocate all files: Checked

Advanced: bt.compact_allocation: false (default)

Advanced: diskio.sparse_files: false (default)

Advanced: diskio.no_zero: true (default) (also tried false, I see no difference)

Hm, a work around is possible, but a little bit of work. How critical is this issue?

Over the span of ten downloaded files, here are my findings: A downloaded file is 1.4GB. When it finished, that file reads at about 3MB/sec to 15MB/sec, (loudly, with much thrashing). After being copied to another drive and back again to its original place, that same test file reads nice and fast (and quietly!) at about 60MB/sec. I find read speed with this:

dd if=/path/test.file of=/dev/null

To answer your question, I hope it's critical enough to get attention.

For all the work that's been done in uTorrent to maximize efficiency, including pre-allocation of files so that data gets effectively written where it's supposed to go, this is a gruesome bug on a clean windshield. A successful work-around would greatly improve the uTorrent experience in Linux. From my hours of searching, it seems many other linux people will appreciate it.

-- Nate

(( Yes, I am aware the kernel caches files in ram. Between tests, I ran this to clear my machine's cache:

dd if=/dev/urandom of=/dev/null bs=1048576 count 2048 Without that, I get ridiculous speeds. ))