So there are a few things going on here.

First, I think we need to report the error code that open_ctree failed with.  I expect that systemd is killing the mount process and that's tripping a mutex_lock_interruptible or something, so the mount fails entirely.

Second, what you're hitting is a known scalability issue in btrfs.  The block groups are read in during mount.  Each block group is 1 GB in size, so you'll have about 76k block groups.  The block group items are located in the extent tree along with all of the extent records, so that means they are ultimately distributed across the disk and are read in with what is effectively a random I/O pattern.  Still, it shouldn't take 5 minutes to do it.

Is there any output in the log for that mount?

Created attachment 812707 [details]
dmesg and journal of unsuccessful boot

Tar archive containing the dmesg and journal output of the unsuccessful boot, when /RAID is enabled in /etc/fstab

Hi Jeff,

thanks for the reply, I try to elaborate as precisely as possible:

I have to add, the RAID is heavily used. We have several millions files on this storage, which is filled up to 70%. The setup is rather easy: Standard openSUSE LEAP 15.1 with NFS and Infiniband being the only additional service that runs on the server.

I started to notice the unresponsive behavior after transferring the old RAID to this one, being consistent with a scalability issue.


Yesterday afternoon the corresponding volume got a `btrfs balance start /RAID`, which triggered the whole filesystem completely unresponsive. A sucessive `btrfs balance status /RAID` was unresponsive for more than an hour without any output (but it has not been killed by the hung task detector either).

Because the whole RAID becomes totally unusable with no visible progress in the last 12h, a `ls -l /RAID` hangs also for more than an hour, I am trying to reboot the machine now. After 5 minutes of not rebooting and now visible activity on any disk either I decided for a forcible reboot (CTRL+ALT+DEL more than 7 times) and another 5 minutes of complete unresponsiveness, no progress and no disk activity I turned the machine off.


After a successful boot (without mounting /RAID) I re-enabled /RAID in /etc/fstab and rebooted :

    ## /etc/fstab
    #UUID=ef07c50a-4759-484c-930d-e8bac9f673e9  /RAID                   btrfs  defaults                      0  0

A move of the whole boot process is at my webserver: ftp://feldspaten.org/dump/20190803_btrfs_balance_issue/btrfs_openctree_failed.mp4

Here, of course, the open_ctree error was appearing. I dumped dmesg and the journal and am attaching both files here. After that, I disabled /RAID in /etc/fstab again and sucessfully rebooted the machine.

After that, I was running a `btrfs device scan` on the unmounted volume:

    eso-storage:~ # time btrfs device scan /dev/sdb1
    Scanning for Btrfs filesystems in '/dev/sdb1'
    
    real	0m0.004s
    user	0m0.001s
    sys	0m0.003s


then I mounted the volume (took 5 minutes)

    eso-storage:~ # time mount /dev/sdb1 /RAID                                                                                
    
    real    5m36.333s
    user    0m0.001s
    sys     0m6.165s

Then, to test the filesystem and actually get an overview what the users do on this system I count the files

    eso-storage:~ # time find /RAID -type f | wc -l

This is gonna take some time. I'll post the output when something changes and restart a `btrfs balance` over the remaining weekend, in the hope the filesystem repairs itself ...

Addendum to the mount. The only messages in dmesg I get after/while mounting are the following:

[  248.218567] BTRFS info (device sdb1): disk space caching is enabled
[  248.218571] BTRFS info (device sdb1): has skinny extents
[  607.781209] BTRFS info (device sda2): qgroup scan completed (inconsistency flag cleared)

Hey there,

The file count operation didn't finish over the weekend. For comparison, I run that one on the old RAID (xfs)

    time find /ESO-RAID -type f | wc -l | tee /root/ESO-count.txt
    18652209
    
    real	0m48.494s
    user	0m18.983s
    sys	0m34.245s

The system itself is pretty busy, load average: 4.00, 4.00, 4.0 (4core Intel Xeon E3-1230). perf top shows the following

    perf top --sort comm,dso
     74.72%  swapper     [kernel]
     25.28%  irqbalance  [kernel]
      0.00%  perf        [kernel]

For us, as we rely on the new NAS we probably need to switch back to xfs.

I could negotiate with my supervisor, that we can use the system this week until Friday to run further tests, if required. So feel free to go all crazy, if you find any use of it :-)

Greetings,
Felix

I could resolve the issue by switching to a different filesystem. Thanks to Jeff for the reply and I hope the issue/log can help make btrfs work better under similar loads.

I'm closing this now because the affected system runs now in a different configuration.