Hi, I'm having some issues with ntfs-3g on a CentOS 5.3 server when running VMWare Server 1.x. The VM guest OS files are stored on a NTFS formatted partition on a drive connected via eSATA. I dont' have any problems mounting the file system and the VM starts up now that I've added mainMem.useNamedFile = "FALSE" to /etc/vmware/config. I can copy large amounts of data from the NTFS file system to an ext3 file system in another partition on the same drive without any problems. In fact, I copied over the entire VM to the ext3 file system and can run it from there without any problems.

The issue with ntfs-3g comes up as soon as I try something that involves a large amount of write traffic in the VM when it's vmdk files are stored in the NTFS file system. After a few moments into let's say tar'ing up a large directory, the VM starts to report errors "mptscsih: ioc0: task abort: SUCCESS". The tar operation grinds to a practical halt and the VM becomes unresponsive. At the same time, from the host OS simple ls commands of the ntfs mount point hang or return incomplete information. Top does not show mount.ntfs-3g using much CPU time during the same timeframe. Eventually the VM tends to recover, though in one case I saw a kernel panic. I do not see any ntfs-3g (or other storage drive) error messages in the host OS during these events.

I tried converting the vmdk files to the pre-allocated format, but still see the same stalling behavior which causes the VM guest OS such issues.

As an additional test, I timed a number of large copy (cp) operations from the guest OS (VM guest directory with 28GB of vmdk files). Copying into a ntfs file system is consistantly about twice as slow as copying into a ext3 file system (all on the same drive). This is true whether the source is ntfs or ext3. During the testing, no error messages were produced with any of the test cases.

This clearly looks like a ntfs-3g write behavior problem/bug. Any suggestions? Work arounds? I'm considering trying Advanced ntfs-3g, but I can't risk this data so I'm not keen on beta type solutions.



The TRN1 directory contains 28GB of data per the du command. The files are large, vmdk files. I'm ignoring the sys time difference as the discrepency should be caused by ntfs-3g being a user space program.

Copy ntfs -> ntfs (top showed mount.ntfs-3g at 55-60% of one CPU core, disk %util 30-40%)
[root@heavy 5.x]# time cp -r TRN1 TRN1_test

real 35m23.703s
user 0m7.818s
sys 2m25.543s

Copy ntfs -> ext3 (top showed mount.ntfs-3g about 15-20% of one CPU core, disk %util 95-100%)
[root@heavy 5.x]# time cp -r TRN1 /mnt2/TRN1_test

real 16m4.635s
user 0m6.046s
sys 5m39.637s

Copy ext3 -> ntfs (top showed mount.ntfs-3g about 60% of one CPU core, disk %util 45-55%)
[root@heavy 5.x]# time cp -r /mnt2/TRN1_test/ .

real 27m4.924s
user 0m7.129s
sys 2m48.105s

Copy ext3 -> ext3 (mount.ntfs-3g not involved, disk %util 95-100%)
[root@heavy mnt2]# time cp -r TRN1/ TRN1_test

real 15m37.439s
user 0m5.124s
sys 4m19.520s



System Information

The server is a HP DL380 with dual 3.20 Zeon CPU's and 6GB of RAM.

[root@heavy /]# uname -a
Linux heavy 2.6.18-128.1.14.el5PAE #1 SMP Wed Jun 17 07:15:54 EDT 2009 i686 i686 i386 GNU/Linux
[root@heavy /]# rpm -qa | grep ntfs
fuse-ntfs-3g-2009.4.4-2.el5.rf
[root@heavy /]# vmware -v
VMware Server 1.0.7 build-108231


[root@heavy /]# mount -v
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/cciss/c0d0p1 on /boot type ext3 (rw)
tmpfs on /tmp type tmpfs (rw,size=2G)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/sda1 on /mnt type fuseblk (rw,allow_other,default_permissions,blksize=4096)
/dev/sda2 on /mnt2 type ext3 (rw)
[root@heavy /]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
91G 36G 51G 42% /
/dev/cciss/c0d0p1 99M 33M 62M 35% /boot
tmpfs 2.0G 36K 2.0G 1% /tmp
tmpfs 2.9G 0 2.9G 0% /dev/shm
/dev/sda1 220G 109G 112G 50% /mnt
/dev/sda2 459G 9.0G 427G 3% /mnt2

ntfs-3g-2009.4.4AR.12 did not fix the scsi device timeouts inside the guest VM OS when running cmds like tar of a large directory.

The results of the same ntfs -> ntfs copy operation did speed up some, but aren't anywhere close to tests that write to ext3. I don't really care about benchmark results, but the timeouts at the guest OS level won't cut it.

[root@heavy 5.x]# time cp -r TRN1 TRN1_test
real 27m11.791s
user 0m7.006s
sys 2m20.703s

Stats during the test:
user 7-9%, sys 12-13%, iowait 8-10%
r/sec 35000, w/sec 35000
disk utilization 40-45%
(cpu usage per top) mount.ntfs-3g 50-55%

The timeout is a vmware problem, you should report it to them.

No, I don't agree. While VMware seems to make this bug appear, I've noticed the same issues doing other tasks.

When you're running a VM on an NTFS partition under Linux using NTFS-3g, you WILL encounter this "hang" problem. During this period, no access to the ntfs-3g volume is available. Everything waits. You can't even do an ls on it. Eventually, access will return, the VM continues and your ls will return results.

It will happen over and over while the VM is doing its thing.

This doesn't happen with any other filesystem. It doesn't even happen if you run a VM from an SMB mount.

So, I believe there's a bug or unsupported feature that VMware tends to reveal. It doesn't mean the bug is in VMware. But nice try passing the buck.

VMware published a knowledge base article about their performance problems. It may help: http://kb.vmware.com/selfservice/micros ... Id=1008885

The commercial Tuxera NTFS kernel driver could certainly improve the performance significantly which is fully multi-threaded but still many people are using happily the open source NTFS-3G driver if VMware is optimally configured.

Regards, Szaka

I am also seeing ntfs-3g slowing down when writing large files.
Sounds like a very similar problem.
I found the issue when trying to clone disk images using ddrescue, I have done this several times now, with the same symptoms each time. The cloning process runs quite quickly at first - about 30MBytes / sec, but as it progresses (o/p file gets larger) it slows down.
I naturally assumed at first it was because I was cloning dodgy drives, but that is not the case, it does the same thing with every drive.
The problem shows up as high CPU usage.
I just recorded ntfs-3g taking 97% of my CPU time when the output file size was about 40Gb, as compared to 66% of cpu when the file was still small. The corresponding transfer rates were 800 kB/s vs 30MB/s
When I moved the outfile onto an ext3 partition the same ddrescue command ran to completion at 60MB/s
I was using a cd recently downloaded from http://www.sysresccd.org/Download
ntfs-3g 2009.4.4AR.21 integrated FUSE 27
XATTRS on, POSIX ACLS off

There are numerous references to similar problems, EG.
http://lists.gnu.org/archive/html/bug-d ... 00004.html
http://lists.gnu.org/archive/html/bug-d ... 00009.html
http://old.nabble.com/eSATA-vs-USB-brid ... 47134.html
http://www.mail-archive.com/bug-ddrescu ... 00424.html
http://forums.macrumors.com/showthread.php?t=462821

I guess this problem is present when writing any large file, but doesn't show up much because most files are too small to show it

Could there be some polynomial function of the file size - eg scanning all index entries for each block written.

Hope this is helpful.

Hi,


This is generally a consequence of disk space being fragmented. Can you collect your file parameters :

gzip the result and attach to your reply.


On current open source ntfs-3g, processing a fragment is proportional to the number of fragments, and total processing of fragments is proportional to the square of the number of fragments, so on a fragmented storage, time may get proportional to the square of the size of the file.

It may be different on Tuxera ntfs which has a different allocation policy for big files.

Regards

Jean-Pierre