Bug 1081022 - VFAT mount option "iocharset=utf8" breaks filename case sensitivity
VFAT mount option "iocharset=utf8" breaks filename case sensitivity
Status: RESOLVED FEATURE
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Kernel
Current
x86-64 openSUSE Factory
: P5 - None : Normal (vote)
: ---
Assigned To: E-mail List
E-mail List
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2018-02-14 15:44 UTC by Stefan Hundhammer
Modified: 2018-03-02 14:11 UTC (History)
3 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
Sample shell session (layout not broken by Bugzilla) (6.32 KB, text/plain)
2018-02-14 15:44 UTC, Stefan Hundhammer
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Stefan Hundhammer 2018-02-14 15:44:00 UTC
Created attachment 760171 [details]
Sample shell session (layout not broken by Bugzilla)

I found this while investigating what mount options to use for bug #1080731:

When a VFAT filesystem is mounted with option "iocharset=utf8", case sensitivity in file/directory names changes its behaviour in weird ways: "mkdir -p" can fail with EEXIST ("mkdir: cannot create directory ‘/mnt/efi’: File exists") which according to the mkdir man page should never happen.

Using mount option "utf8" instead which should be the same according to the "mount" man page changes the behaviour to normal.

For one thing, "mkdir -p" should never fail with EEXIST. But that's a matter of that tool which might do the check if a directory already exists the wrong way. That's not what this bug is all about.

The weird behaviour is that this check behaves inconsistently depending on the mount options used: "iocharset=utf8" should not change the way at all that a purely ASCII file/directory name is treated. There are no non-ASCII characters involved at all.

And there appears to be a difference if a VFAT "long name" exists or not: If there is only a "short name" (the old MS-DOS 8+3 filename), the bug appears. If there is also a "long name", everything works fine.


In addition to that, the promise that mount option "utf8" is equivalent to "iocharset=utf8" is broken: Obviously, they behave differently.


Sample shell session (also attached):

[root @ morgul] ~ # uname -a
Linux morgul 4.14.9-1-default #1 SMP PREEMPT Mon Dec 25 15:42:48 UTC 2017 (9423ca2) x86_64 x86_64 x86_64 GNU/Linux

#
# Using iocharset=utf8
#
# FAIL
#

[root @ morgul] ~ # mkfs.vfat /dev/sdb1                                     
mkfs.fat 4.1 (2017-01-24)
[root @ morgul] ~ # mount -t vfat -o iocharset=utf8 /dev/sdb1 /mnt
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
mkdir: cannot create directory ‘/mnt/efi’: File exists
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
mkdir: cannot create directory ‘/mnt/efi’: File exists
[root @ morgul] ~ # grep sdb /proc/mounts
/dev/sdb1 /mnt vfat rw,relatime,fmask=0002,dmask=0002,allow_utime=0020,codepage=437,iocharset=utf8,shortname=mixed,errors=remount-ro 0 0
[root @ morgul] ~ # umount /dev/sdb1


# OK

[root @ morgul] ~ # mkfs.vfat /dev/sdb1
mkfs.fat 4.1 (2017-01-24)
[root @ morgul] ~ # mount -t vfat -o iocharset=utf8 /dev/sdb1 /mnt
[root @ morgul] ~ # grep sdb /proc/mounts
/dev/sdb1 /mnt vfat rw,relatime,fmask=0002,dmask=0002,allow_utime=0020,codepage=437,iocharset=utf8,shortname=mixed,errors=remount-ro 0 0
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # umount /dev/sdb1

#
# Using default mount options (i.e. implicitly iocharset=iso8859-1)
#
# OK

[root @ morgul] ~ # mkfs.vfat /dev/sdb1   
mkfs.fat 4.1 (2017-01-24)
[root @ morgul] ~ # mount -t vfat /dev/sdb1 /mnt                  
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
[root @ morgul] ~ # grep sdb /proc/mounts
/dev/sdb1 /mnt vfat rw,relatime,fmask=0002,dmask=0002,allow_utime=0020,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 0 0
[root @ morgul] ~ # umount /dev/sdb1


# OK

[root @ morgul] ~ # mkfs.vfat /dev/sdb1   
mkfs.fat 4.1 (2017-01-24)
[root @ morgul] ~ # mount -t vfat /dev/sdb1 /mnt
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # grep sdb /proc/mounts
/dev/sdb1 /mnt vfat rw,relatime,fmask=0002,dmask=0002,allow_utime=0020,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 0 0
[root @ morgul] ~ # umount /dev/sdb1

#
# Using iocharset=iso8859-15
#
# OK

[root @ morgul] ~ # mkfs.vfat /dev/sdb1   
mkfs.fat 4.1 (2017-01-24)
[root @ morgul] ~ # mount -t vfat -o iocharset=iso8859-15 /dev/sdb1 /mnt
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
[root @ morgul] ~ # grep sdb /proc/mounts
/dev/sdb1 /mnt vfat rw,relatime,fmask=0002,dmask=0002,allow_utime=0020,codepage=437,iocharset=iso8859-15,shortname=mixed,errors=remount-ro 0 0
[root @ morgul] ~ # umount /dev/sdb1


# OK

[root @ morgul] ~ # mkfs.vfat /dev/sdb1
mkfs.fat 4.1 (2017-01-24)
[root @ morgul] ~ # mount -t vfat -o iocharset=iso8859-15 /dev/sdb1 /mnt
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # grep sdb /proc/mounts
/dev/sdb1 /mnt vfat rw,relatime,fmask=0002,dmask=0002,allow_utime=0020,codepage=437,iocharset=iso8859-15,shortname=mixed,errors=remount-ro 0 0
[root @ morgul] ~ # umount /dev/sdb1

#
# Using utf8 (which should be equivalent to iocharset=utf8)
#
# OK

[root @ morgul] ~ # mkfs.vfat /dev/sdb1   
mkfs.fat 4.1 (2017-01-24)
[root @ morgul] ~ # mount -t vfat -o utf8 /dev/sdb1 /mnt
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
[root @ morgul] ~ # grep sdb /proc/mounts
/dev/sdb1 /mnt vfat rw,relatime,fmask=0002,dmask=0002,allow_utime=0020,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro 0 0
[root @ morgul] ~ # umount /dev/sdb1


# OK

[root @ morgul] ~ # mkfs.vfat /dev/sdb1
mkfs.fat 4.1 (2017-01-24)
[root @ morgul] ~ # mount -t vfat -o utf8 /dev/sdb1 /mnt
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # grep sdb /proc/mounts
/dev/sdb1 /mnt vfat rw,relatime,fmask=0002,dmask=0002,allow_utime=0020,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro 0 0
[root @ morgul] ~ # umount /dev/sdb1


# Using utf8 and iocharset=utf8 combined
#
# FAIL
#

[root @ morgul] ~ # mkfs.vfat /dev/sdb1
mkfs.fat 4.1 (2017-01-24)
[root @ morgul] ~ # mount -t vfat -o utf8,iocharset=utf8 /dev/sdb1 /mnt
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
mkdir: cannot create directory ‘/mnt/efi’: File exists
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
mkdir: cannot create directory ‘/mnt/efi’: File exists
[root @ morgul] ~ # umount /dev/sdb1


# OK

[root @ morgul] ~ # mkfs.vfat /dev/sdb1
mkfs.fat 4.1 (2017-01-24)
[root @ morgul] ~ # mount -t vfat -o utf8,iocharset=utf8 /dev/sdb1 /mnt
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # grep sdb /proc/mounts
/dev/sdb1 /mnt vfat rw,relatime,fmask=0002,dmask=0002,allow_utime=0020,codepage=437,iocharset=utf8,shortname=mixed,utf8,errors=remount-ro 0 0
[root @ morgul] ~ # umount /dev/sdb1


# FAIL

[root @ morgul] ~ # mkfs.vfat /dev/sdb1
mkfs.fat 4.1 (2017-01-24)
[root @ morgul] ~ # mount -t vfat -o iocharset=utf8,utf8 /dev/sdb1 /mnt
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
mkdir: cannot create directory ‘/mnt/efi’: File exists
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
mkdir: cannot create directory ‘/mnt/efi’: File exists
[root @ morgul] ~ # grep sdb /proc/mounts
/dev/sdb1 /mnt vfat rw,relatime,fmask=0002,dmask=0002,allow_utime=0020,codepage=437,iocharset=utf8,shortname=mixed,utf8,errors=remount-ro 0 0
[root @ morgul] ~ # umount /dev/sdb1
Comment 1 Stefan Hundhammer 2018-02-14 16:37:11 UTC
I also tried with additional "shortname=" options in all combinations, but that doesn't change very much:


[root @ morgul] ~ # mkfs.vfat /dev/sdb1
mkfs.fat 4.1 (2017-01-24)
[root @ morgul] ~ # mount -t vfat -o iocharset=utf8,shortname=win95 /dev/sdb1 /mnt
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
mkdir: cannot create directory ‘/mnt/efi’: File exists
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
mkdir: cannot create directory ‘/mnt/efi’: File exists
[root @ morgul] ~ # umount /dev/sdb1
[root @ morgul] ~ # mount -t vfat -o iocharset=utf8,shortname=lower /dev/sdb1 /mnt
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
mkdir: cannot create directory ‘/mnt/EFI’: File exists
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
mkdir: cannot create directory ‘/mnt/EFI’: File exists
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
[root @ morgul] ~ # umount /dev/sdb1
[root @ morgul] ~ # mount -t vfat -o iocharset=utf8,shortname=winnt /dev/sdb1 /mnt
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # mkdir -p /mnt/EFI/boot
[root @ morgul] ~ # mkdir -p /mnt/efi/boot
mkdir: cannot create directory ‘/mnt/efi’: File exists


With "shortname=lower" it just reverses when the error appears, all other cases remain the same.
Comment 2 Takashi Iwai 2018-02-15 06:01:26 UTC
The UTF-8 handling in vfat looks really confusing.  The iocharset=utf8 seems still like a non-recommended option (and the kernel warns you, too).  The documentation tells that it results in the case-sensitive problem, which is likely what you saw.

Meanwhile, "utf8" mount option (or "utf8=true") while keeping isocharset as default would allow UTF-8 in a different way.  Does this work better?
Comment 3 Stefan Hundhammer 2018-02-15 08:47:03 UTC
(In reply to Takashi Iwai from comment #2)
> The UTF-8 handling in vfat looks really confusing.  The iocharset=utf8 seems
> still like a non-recommended option (and the kernel warns you, too).  The
> documentation tells that it results in the case-sensitive problem, which is
> likely what you saw.
> 
> Meanwhile, "utf8" mount option (or "utf8=true") while keeping isocharset as
> default would allow UTF-8 in a different way.  Does this work better?

Yes, using "utf8" instead of "iocharset=utf8" does not show this problem - see shell session above. But IIRC the "mount" man page explicitly says that this is deprecated, and "iocharset=utf8" should be used instead.

It is also unclear what's the difference between those two; I couldn't find any hint. I understood that they should be equivalent, but obviously, they are not; at least not in this context.
Comment 4 Takashi Iwai 2018-02-15 09:25:31 UTC
(In reply to Stefan Hundhammer from comment #3)
> (In reply to Takashi Iwai from comment #2)
> > The UTF-8 handling in vfat looks really confusing.  The iocharset=utf8 seems
> > still like a non-recommended option (and the kernel warns you, too).  The
> > documentation tells that it results in the case-sensitive problem, which is
> > likely what you saw.
> > 
> > Meanwhile, "utf8" mount option (or "utf8=true") while keeping isocharset as
> > default would allow UTF-8 in a different way.  Does this work better?
> 
> Yes, using "utf8" instead of "iocharset=utf8" does not show this problem -
> see shell session above.

Ah I overlooked it, thanks.

> But IIRC the "mount" man page explicitly says that
> this is deprecated, and "iocharset=utf8" should be used instead.

Hm, I couldn't find that description in man page on TW...

OTOH, the kernel warning with iocharset=utf8 is still valid with the
latest upstream kernel.

> It is also unclear what's the difference between those two; I couldn't find
> any hint. I understood that they should be equivalent, but obviously, they
> are not; at least not in this context.

They should serve equivalently but the UTF-8 translation is applied
differently.  With iocharset=utf8, the translation is done via NLS
layer, which is outside the vfat driver.  Meanwhile, with utf8=true,
the vfat driver itself performs the UTF-8 translation.
Comment 5 Stefan Hundhammer 2018-02-19 15:55:48 UTC
I just double-checked, and I can't find that bit about "utf8=true" being deprecated anymore. I might have read it online in a man page that is not relevant to us.

So, you say   

  s/iocharset=utf8/utf8=true/g    

should generally be a workaround for this problem? That would work for me.
Comment 6 Takashi Iwai 2018-02-19 16:00:18 UTC
(In reply to Stefan Hundhammer from comment #5)
> So, you say   
> 
>   s/iocharset=utf8/utf8=true/g    
> 
> should generally be a workaround for this problem? That would work for me.

Yes, that's my understanding, at least for VFAT.
It should be enough simply to pass "utf8", too.
Comment 7 Takashi Iwai 2018-03-02 14:11:17 UTC
Let's close as FEATURE.  Not ideal, but it's life.