I'm pretty convinced that this isn't a NTFS-3G problem, but this is where I came first, so I'm posting this here to try and help anyone else who runs into this problem.

I got a new laptop, installed Vista64, threw together a 1TB external USB with an old enclosure I had, formatted it as NTFS under Vista64, all was well. Plugged it into a Linux box and copied some files over, and everything went fine.

Tried plugging it back into Vista64 and it claims it was corrupt. Disk Management says it was RAW, but manually running chkdsk in the Vista64 installed showed it was NTFS, with 0 errors.

Plugged it back into Linux, recognised, and all my files in tact. Plugged it into a Windows XP install, recognised, and all my files in tact.

Ran ntfsfix on the Linux box and chkdsk on the Windows XP install, neither reported any problems. Tried in Vista64 again, and, it still claims its RAW, even though its own chkdsk can read it just fine.

Just to check if it wasn't a configuration or driver issue, or something else specific to my Vista64 machine, I installed Windows XP on it, and, it worked fine and could see all my files, and then I reinstalled Vista64, but it still couldn't see the files.

Just to be extra sure, I installed Vista64 on a completely different machine, and that couldn't see the files either.

And then for completely unrelated issues (Mainly software and hardware support for Vista64 sucking) I installed a 32 bit version of Vista and was quite surprised to find it worked flawlessly.
To double check I installed 32 bit Vista on the completely separate machine I put Vista64 on previously, and it worked on that, also.

So, two separate machines that flat out refused to read an ext. HDD while running clean installs of Vista64, can access it fine in Vista x86...

I've been reading and writing with ntfs-3g on the linux box all day, and it still works fine on the two Vista x86 installs.

Googling I found a lot of people with similar problems, and a lot of them were using 64bit Vista, and the rest didn't specify which version they were using, and none of them came back saying "Fixed it!"
It seems most people resorted to formatting the drive and not using it under Linux, but that wasn't an option for me, but luckily switching to 32bit Vista seems to have done the trick...

If this doesn't belong here feel free to delete it, I just posted it here because, like I said, this was the first place that I myself came to when looking for a solution.

Hi,

Thank you for sharing your experiences. It was very interesting. This is what we know:

1. Vista64 has a lot of reliability problems, crashes and corruptions. Even if people never used NTFS-3G, they have similar problems. 64-bit computing is very new for Microsoft and for most of their developer partners (writing the device drivers, etc). Apparently their code is not quite ready yet for 64-bit production use.

2. Despite the above point, quite a lot of people use Vista64 and NTFS-3G together successfully. I'm one of them.

3. The RAW file system often means partition table, boot sector corruption or device driver problems (USB , SATA, eSATA, etc). It could be that Linux changes some hardware settings and the relevant Vista64 device driver doesn't or can not reset it. If you can reformat the device then of course this is out of question.

The problem seems to be not on the file system but in some other level on Vista64. Your tests also confirm this when things just work fine on XP and Vista x86.

Regards, Szaka

So you installed Windows 6 times just to sort the issue? You sir, are a martyr!

Thanks for sharing.

I can confirm this problem on Vista32. I don't think this is a Vista64 issue, I have reproduced the exact symptoms on 3 different Vista32 machines. They all refuse to work on a drive that has been used with NTFS-3g. Been racking my head for a while to sort this out.

You must have something unique installed on either Vista or Linux which breaks the file system accessibility on Vista. NTFS-3G is not so unique because it is daily used by millions of people, device.

We would be very interested to know if you can figure out something useful. Meanwhile we will note your problem to Microsoft and ask them to show the errors with more details for troubleshooting.

I seem to have a very similar problem. I have a 1TB external harddrive with a 100GB ext3 and a 830GB ntfs-partition. Since about a year I cannot access the ntfs volume from my Laptop running Vista32 Ultimate (it shows up as raw), but have no problems accessing it from any XP or Linux machine, even from the same Laptop, which cannot access it using Vista. Even moving and shrinking the whole ntfs partition using gparted by a few tens of GB didn't change anything with that situation. It was even possible to make a new ntfs partition in the freed space, which is nicely visible from Vista albeit not the original one. It is very ironic, that I can easily access the ext3 partition existing besides the ntfs partition from vista using ifs - I'd like to have ntfs-3g for vista. Now, that I have copied all the files in the bogus partition to a new harddrive and are ready to repartition it to get read of the problem, is there any feasible way to diagnose the problem before I do that to avoid it in the future?

Hi,


Vista may have some unknown requirement (wanted or unwanted). On the gparted website there has been an unsolved case for three years about Vista not booting after a partition resizing.
What I would do, is to use Vista to format the partition again, you would get a better chance to meet Vista requirements.

You can save your partition parameters by ntfsclone for later examination, but expertise on Vista requirements is needed for the examination to be useful.

Regards

Jean-Pierre

Thank you Jean-Pierre for the suggestions.
I managed to delete all the files and shrink the partition in question to 3GB using gparted. Afterwards I produced a compressed clone file with ntfsclone and bzip2, which now has a size of only 1.6MB.
Surprisingly, the phenomenology of the filesystem in question survived the whole ordeal and remains the same, even after I restore it from that 1.6MB compressed image: It is readily accessible from ntfs-3g and XP, but not from Vista. I.e., the problem is indeed perfectly reproducible. Is there somebody I could send these 1.6MB to?

Hi,


You really should format by Vista, you could get a useful error message.
Did you try to put the partition contents elsewhere ? There might be some constraint in the partition table for Vista (not after an ext3 partition, must start at some sector alignment, etc).

Either you upload it to some public serveur, or you send a "PM" to me and I reply with an email address where you can send the compressed data.
But apart from trying to mount your image into my own Vista and checking for unusual formatting parameters, I will probably not be of much help...

Regards

Jean-Pierre

I did not try to reformat the bogus partition itself, as the problem seems to be connected to the ntfs image, and not the partition, since I verified it being reproducible on two different partitions using the restore function of ntfsclone.
Gparted also normally doesn't produce errors with vista, which I verified by shifting yet another - this time vista-readable - ntfs partition around, which remained Vista-readable after that process.

I really am interested in getting behind this problem, so I'll send you a PM in a minute, thanks!

Hi,

I have been able to mount your partition on Vista (see picture)

First your MFT (master file table) is very fragmented : it has 25 fragments. I defragmented it on Win XP, but it could still not mount on Vista.

Then I saw that your root directory has two unusual attributes : an object id, and an attribute unknown to me, code 0x100 and name "$TXF_DATA". I removed those attributes and it mounted. However it took Vista one minute or two to mount, it has probably made a chkdsk before mounting.

The cause was probably this attribute "$TXF_DATA". Only you can tell on what circumstance you created that (by activating some transactional process ?) The object id is more common, but I had not seen it on the root directory so far.

Regards

Jean-Pierre

That is fantastic that you managed to do that! I have given up completely (and reformatted).
But what did you do to see and delete this bogus attribute (should it ever happen again)?
(I didn't do anything to my harddisk except changing partition size with gparted. Somehow vista must have done it by herself.)
Another thing: How can I defragment just the MFT and not the individual files?

Related to TXF_DATA I found a error note at microsoft.com:
http://support.microsoft.com/kb/974729
(This doen't seem to load, but another version is here:
http://209.85.229.132/search?q=cache:S3 ... en|lang_de
)
So it seems it is known, but one has to know first, what to search for.

Hi,


Well, you had deleted nearly all your files, leaving few data for me to analyze (my first aim being to check for viruses), and I quickly spotted the strange attribute. I have made a "quick and dirty" patch to ntfs-3g to delete it.

Associated to the attribute are a few hidden files :

These file names lead me to think of some mirroring software having been installed (probably on the 26th August 2008).

It would be very useful you remember what this is about, because the only explanation I found (http://book.51cto.com/art/200911/160841.htm) is not much clear.

Microsoft (in the article you referenced) documents Vista as being unable in some conditions to mount volumes which contain the said extension, and puts the blame on Linux for having made the extension non resident.

Now the problem is : should ntfs-3g take care of not making the extension non resident, or should it refuse to mount the volume read-write if the extension is present ? I tend to think the second option is the right one, because any modification made by ntfs-3g would not agree with what the unknown software would expect, which might turn into data loss when synchronizing (if this is the purpose).

So, can you remember what this software is about, or what you installed on the said date ?


I just used the standard defragmenter on WinXP. It may have just truncated the MFT, which was possible because the files were deleted.

Regards

Jean-Pierre

The date can be explained: I got the harddisk on 26th August 2008.
Consciously, I never installed any programs on the external hard drive - I just used it for long term storage of files, which I copied by hand in windows or linux. (As with the 3 other external hard drives I own, which do not show any such problem.)
I also do not use any mirroring or backup software in windows.

I do not see any of these TXF_DATA files on my other drives.
I am very curious - you must have done something special, as I don't remember seeing them when I last looked at my sick drive with ls -la under ntfs-3g or in the explorer with all hidden files being shown:
What exactly do I have to do to list these hidden files?
Maybe I can find this attribute on the other disks as well and track down where it stem from ...

Hi,


With Linux you can list them normally, just escape the '$' so that it is not taken as an environment variable. You will not be allowed to make a modification or display the contents.

The "$TXF_DATA" is an attribute name. You can display the details of the root directory by issuing the command :



That is strange. Did you use Vista to format the partition initially ? It could be a feature in some Vista configuration.
Microsoft apparently considers the mount refusal as a bug in Vista. There is a fix, however it is not available in Windows Update. I made an update yesterday, and I still have the old files.

The very problem is to identify whether mounting such volumes read-write on Linux should be allowed. I do not know whether the transactional link is permanent or temporary. If it is permanent, writing without logging the transaction records may result in valuable data loss, so writing should be forbidden. So please report on your findings.

Regards

Jean-Pierre

This issue has been fixed in NTFS-3G 2010.1.16.

The fundamental problem is that the latest Microsoft NTFS implementations ignore the on-disk NTFS specifications, namely the $AttrDef file allowing the $LOGGED_UTILITY_STREAM:TXF_DATA named attribute to be non-resident, and refuse to interpret a file, directory or even volume if this is the case.

Thanks a lot for the highly valuable help and investigation!

Regards, Szaka

Dear szaka, thank you for this information. Does this mean, the erroneous attribute has been somehow set in the past by accessing the partition with NTFS-3G? As mentioned I certainly have never set it consciously.

Hi,


Just mounting on Vista creates the attribute, but so far we have been able to get ntfs-3g to make it non-resident only through very specific procedures. We have no evidence it was made non-resident by ntfs-3g.

Note that mounting with the latest advanced ntfs-3g makes the attribute resident, and the partition mountable by Vista again.

Regards

Jean-Pierre

It could be. We could reproduce the case with NTFS-3G but it was very difficult and unlikely usage scenario. It very well could be that other software (on Windows, Linux, etc) also have/had this problem.

Basically the latest Vista and newer Microsoft NTFS drivers don't follow strictly enough the on-disk NTFS specification and this can/could lead problems with all previous NTFS drivers (Windows, Linux, etc). I believe this was not intentional, just an oversight. NTFS is very complex and it's easy to miss one of the many hairy details.

Fantastic. Thank you all. Now I do not have any worries about this being a problem in the future!

I have an external hard drive with LOTS of music on it that is now reading file system raw with 0 used space and 0 free space.
It is a cavalry CAUI35160....160 gb

I could not access it and on further inspection I find in properties what i mentioned
0 used space and 0 free space
File system:RAW

I cant find anywhere what the file system was to begin with,never felt the need to check.

I have found instructions on how to change the file system back to its original form and keep the data by using a program on partition magic and making 2 floppy disk to boot the system in PURE dos mode and changing the file system on the correct drive. BUT I have to know 100% sure of the original file system to change back to. IE fat 32,NTFS or some other....I am sure it is one of the later but how do i find out what it was from the manufacture.

Do all external drives come in FAT32 ? I notice all the boards i read people have said that there drives came in fat32 format,and wonder if this will cause them probs.but that is as close as i can come to finding out what this drives format was originaly.
Cavalry 160GB external hard drive CAUI35160
Any help would be appreciated
Please

Hi,


Which operating system are you using ? After using which software were you not able to read files from this drive any more .

Regards

Jean-Pierre