Bug 1111991 - No boot after kernelupdate to 4.12.14-lp150.12.19.1
No boot after kernelupdate to 4.12.14-lp150.12.19.1
Status: RESOLVED NORESPONSE
Classification: openSUSE
Product: openSUSE Distribution
Classification: openSUSE
Component: Kernel
Leap 15.0
x86-64 SUSE Other
: P5 - None : Major (vote)
: ---
Assigned To: Wenruo Qu
E-mail List
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2018-10-16 11:01 UTC by Tom Hanker
Modified: 2019-02-25 04:58 UTC (History)
3 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
btrfs --check (15.28 KB, application/vnd.oasis.opendocument.text)
2018-10-19 06:02 UTC, Tom Hanker
Details
btrfs check output, in text format. (8.68 KB, text/plain)
2018-10-23 17:05 UTC, Jeff Mahoney
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Tom Hanker 2018-10-16 11:01:30 UTC
User-Agent:       Mozilla/5.0 (X11; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0
Build Identifier: 

After Update through zypper, i can´ boot anymore. After starting Grub the error occur something like: btrfs critical btrfs corrupted (sda2) leaf........

then the system try to start services like modem-manager..., but it failled. I dit a rollback and with kernel 4.12.14-lp150.12.16, all went smoothly. 

Reproducible: Always
Comment 1 Andreas Stieger 2018-10-16 11:19:03 UTC
Are you using VirtualBox? (bug 1111076)
Does 4.12.14-lp150.12.22.1 fix the issue for you?
Comment 2 Tom Hanker 2018-10-16 13:13:14 UTC
No i am not using Virtualbox, i use kvm. I haven´t installed any virtualbox package. I also tried to uninstall the nvidia driver, but no change. I wonder how could be btrfs damaged with kernel ...19 but not with .19 and .22. Meanwhile i tried the kernel .22 the same behaviour than with .19.
Comment 3 Takashi Iwai 2018-10-16 13:37:56 UTC
Then it must be something else.

Could you try to get the whole kernel boot messages via serial console with the latest kernel (after backing up the VM)?

Since it sounds like a btrfs issue, reassign to Jeff for now.
Comment 4 Jeff Mahoney 2018-10-16 13:39:40 UTC
This looks like the tree checker may be a bit overzealous.
Comment 5 Wenruo Qu 2018-10-16 13:43:44 UTC
Would you please paste the kernel message? At least full line of the corrupted leaf part.

It's indeed tree-checker causing the problem, but normally tree-checker error is really showing something wrong.

And in that case, I also recommend to use some rescue usb to boot and run latest btrfs check on the fs.

Thanks,
Qu
Comment 6 Tom Hanker 2018-10-16 13:59:11 UTC
(In reply to Wenruo Qu from comment #5)
> Would you please paste the kernel message? At least full line of the
> corrupted leaf part.
> 
> It's indeed tree-checker causing the problem, but normally tree-checker
> error is really showing something wrong.
> 
> And in that case, I also recommend to use some rescue usb to boot and run
> latest btrfs check on the fs.
> 
> Thanks,
> Qu

I can´t get a kernel message, the system doesn´t boot. Is there a way to get the messages?
Comment 7 Takashi Iwai 2018-10-16 14:02:03 UTC
You can set up a serial console for VM and pass console=ttyS0 boot option.
Also try to remove quiet and splash=xxx boot options, too.
Comment 8 Wenruo Qu 2018-10-16 14:03:07 UTC
(In reply to Tom Hanker from comment #6)
> (In reply to Wenruo Qu from comment #5)
> > Would you please paste the kernel message? At least full line of the
> > corrupted leaf part.
> > 
> > It's indeed tree-checker causing the problem, but normally tree-checker
> > error is really showing something wrong.
> > 
> > And in that case, I also recommend to use some rescue usb to boot and run
> > latest btrfs check on the fs.
> 
> I can´t get a kernel message, the system doesn´t boot. Is there a way to get
> the messages?

You could use rescue USB to boot into some other distro, and run btrfs check on the fs, then try to mount it.

It's recommended to use latest upstream kernel and latest btrfs-progs first. You can use Archlinux iso to do it.

Thanks,
Qu
Comment 9 Tom Hanker 2018-10-16 16:30:27 UTC
(In reply to Wenruo Qu from comment #8)
> (In reply to Tom Hanker from comment #6)
> > (In reply to Wenruo Qu from comment #5)
> > > Would you please paste the kernel message? At least full line of the
> > > corrupted leaf part.
> > > 
> > > It's indeed tree-checker causing the problem, but normally tree-checker
> > > error is really showing something wrong.
> > > 
> > > And in that case, I also recommend to use some rescue usb to boot and run
> > > latest btrfs check on the fs.
> > 
> > I can´t get a kernel message, the system doesn´t boot. Is there a way to get
> > the messages?
> 
> You could use rescue USB to boot into some other distro, and run btrfs check
> on the fs, then try to mount it.
> 
> It's recommended to use latest upstream kernel and latest btrfs-progs first.
> You can use Archlinux iso to do it.
> 
> Thanks,
> Qu

What Distor can i use? Ubuntu? I haven´t Arch Linux but i can download it on Multisystem. Has Ubuntu and ARchlinux the tools to start btrfs check /dev/sda2. 
Or should i use an Opensuse LiveCD? I will try it tomorrow. But if the filesystem is broken in kerne .19, why not in kernel .16? The Kernel .16 works without Problems. By the way the System isn´t installed on a VirtualMachine.
Comment 11 Jeff Mahoney 2018-10-16 17:45:44 UTC
You don't need to use another distro.  The openSUSE Tumbleweed rescue ISO has the latest kernel and tools:

http://download.opensuse.org/tumbleweed/iso/openSUSE-Tumbleweed-Rescue-CD-x86_64-Current.iso

But before you do that, we'd be interested in the messages the kernel is producing now.  Takashi's suggestion to capture the kernel log using the virtual serial console and without 'quiet' is your best bet.
Comment 12 Tom Hanker 2018-10-16 19:09:51 UTC
(In reply to Jeff Mahoney from comment #11)
> You don't need to use another distro.  The openSUSE Tumbleweed rescue ISO
> has the latest kernel and tools:
> 
> http://download.opensuse.org/tumbleweed/iso/openSUSE-Tumbleweed-Rescue-CD-
> x86_64-Current.iso
> 
> But before you do that, we'd be interested in the messages the kernel is
> producing now.  Takashi's suggestion to capture the kernel log using the
> virtual serial console and without 'quiet' is your best bet.

I don´t know anything about a "virtual serial console" I red the command btrfs check --repair could take about 24 hours.
Comment 13 Takashi Iwai 2018-10-16 20:27:10 UTC
How do you start your VM?  If you're using some GUI for managing VM, there must be a configuration menu to allow user to enable a serial-console.  After enabling it, start the VM and pass "console=ttyS0" boot option.  Then the kernel messages will be output to that serial console.

If you start from a command-line, passing "-serial stdio" to qemu-kvm will lead the serial output to stdout, for example.
Comment 14 Tom Hanker 2018-10-17 09:52:08 UTC
(In reply to Takashi Iwai from comment #13)
> How do you start your VM?  If you're using some GUI for managing VM, there
> must be a configuration menu to allow user to enable a serial-console. 
> After enabling it, start the VM and pass "console=ttyS0" boot option.  Then
> the kernel messages will be output to that serial console.
> 
> If you start from a command-line, passing "-serial stdio" to qemu-kvm will
> lead the serial output to stdout, for example.

I usw virtmanager. So i add a device. In this case serial (pseudo TTY pty). Or should i add a console pseudo TTY pty typ virt0? I did it then in menu disply textconsole i can see serial1 and of course there isn´t any text. If i put the line GRUB_CMDLINE_LINUX_DEFAULT=”console=ttyS0″ to /etc/default/grub, then ei can read the messages in the textconsole serial1?
Comment 15 Takashi Iwai 2018-10-17 10:49:00 UTC
Just add console=ttyS0 should work.  You can add it temporarily in GRUB menu, or permanently via /etc/default/grub.

Please refer to the guide
  https://doc.opensuse.org/documentation/leap/virtualization/html/book.virt/book.virt.html
Comment 16 Tom Hanker 2018-10-18 09:50:04 UTC
The next Problem. i wanted to add in virtualmanager a device. serial - pseudo TTY (pty), then i got this measage.

internal error: Kann Qemu-Befehl 'device_add' nicht ausführen: Bus 'isa.0' does not support hotplugging

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/addhardware.py", line 1318, in _add_device
    self.vm.attach_device(self._dev)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1093, in attach_device
    self._backend.attachDevice(devxml)
  File "/usr/lib64/python3.6/site-packages/libvirt.py", line 570, in attachDevice
    if ret == -1: raise libvirtError ('virDomainAttachDevice() failed', dom=self)
libvirt.libvirtError: internal error: Kann Qemu-Befehl 'device_add' nicht ausführen: Bus 'isa.0' does not support hotplugging
Comment 17 Tom Hanker 2018-10-18 10:12:19 UTC
I managed to add a serial console, i edited /etc/boot/default, make a restart and boot with kernel .19. The usual errors occur. Then i restart boot in kernel .16, open the virtmanager, but the serialconsole has no messeges. but i could found messeges in /var/log/boot. 

[  OK  ] Started Check if mainboard battery is Ok.
[  OK  ] Started Rotate log files.
[  OK  ] Started Backup /etc/sysconfig directory.
         Starting Locale Service...
[  OK  ] Started Apply settings from /etc/sysconfig/keyboard.
[  OK  ] Started Save/Restore Sound Card State.
[  OK  ] Started Name Service Cache Daemon.
[FAILED] Failed to start Virtual Machine and Container Registration Service.
See 'systemctl status systemd-machined.service' for details.
[  OK  ] Started Machine Check Exception Logging Daemon.
[  OK  ] Started Update system wide CA certificates.
[  OK  ] Started Initialize hardware monitoring sensors.
[FAILED] Failed to start Modem Manager.
See 'systemctl status ModemManager.service' for details.
[FAILED] Failed to start wicked AutoIPv4 supplicant service.
See 'systemctl status wickedd-auto4.service' for details.
[FAILED] Failed to start firewalld - dynamic firewall daemon.
See 'systemctl status firewalld.service' for details.
[FAILED] Failed to start wicked DHCPv4 supplicant service.
See 'systemctl status wickedd-dhcp4.service' for details.
[FAILED] Failed to start wicked DHCPv6 supplicant service.
See 'systemctl status wickedd-dhcp6.service' for details.
[FAILED] Failed to start Avahi mDNS/DNS-SD Stack.
See 'systemctl status avahi-daemon.service' for details.
[FAILED] Failed to start LSB: Supports the direct execution of binary formats..
See 'systemctl status jexec.service' for details.

Thsi are the errors when i boot with kernel .19 But i can´t find the messeges about the btrfs error.
Comment 18 Takashi Iwai 2018-10-18 10:21:59 UTC
Please remove "quiet" and "splash=xxx" boot options.  Then you must see lots of messages in serial console.
Comment 19 Tom Hanker 2018-10-18 13:19:46 UTC
(In reply to Takashi Iwai from comment #18)
> Please remove "quiet" and "splash=xxx" boot options.  Then you must see lots
> of messages in serial console.

I did, but can´t see any message in the console. I only can see the detailed messeges on the boot. I found a similar output: 

BTRFS critical (device sda2): corrupt leaf, slot offset bad: block=43231330304,root=1, slot=47

This message comes 3 times, then the it goes on with the message above. I red it could be a hardware crash, but if why not with kernel .16
Comment 20 Wenruo Qu 2018-10-18 13:27:56 UTC
> BTRFS critical (device sda2): corrupt leaf, slot offset bad:
> block=43231330304,root=1, slot=47
> 

That's what we need.

It's a corruption in root tree, so it's really serious.

We need some extra info about the corruption.

Would you please use the following command to dump that corrupted block?

# btrfs ins dump-tree -b 43231330304 /dev/sda2

> This message comes 3 times, then the it goes on with the message above. I
> red it could be a hardware crash, but if why not with kernel .16

Because .16 kernel doesn't has such restrict content check.
But strangely, the error message doesn't provide enough info, unlike latest upstream.
(Normally it should provide more detailed info on which is going wrong).

Anyway, that "btrfs ins dump-tree" should tell us if it's just a minor fixable problem or a real corruption.

Thanks,
Qu
Comment 21 Tom Hanker 2018-10-18 13:50:43 UTC
(In reply to Wenruo Qu from comment #20)
> > BTRFS critical (device sda2): corrupt leaf, slot offset bad:
> > block=43231330304,root=1, slot=47
> > 
> 
> That's what we need.
> 
> It's a corruption in root tree, so it's really serious.
> 
> We need some extra info about the corruption.
> 
> Would you please use the following command to dump that corrupted block?
> 
> # btrfs ins dump-tree -b 43231330304 /dev/sda2
> 
> > This message comes 3 times, then the it goes on with the message above. I
> > red it could be a hardware crash, but if why not with kernel .16
> 
> Because .16 kernel doesn't has such restrict content check.
> But strangely, the error message doesn't provide enough info, unlike latest
> upstream.
> (Normally it should provide more detailed info on which is going wrong).
> 
> Anyway, that "btrfs ins dump-tree" should tell us if it's just a minor
> fixable problem or a real corruption.
> 
> Thanks,
> Qu

Ok. the number 43231330304 was only an example, which i copy from the net. i bootet again and got the right one. this is the output, very long:

btrfs ins dump-tree -b 301168066560 /dev/sda2
btrfs-progs v4.15
leaf 301168066560 items 230 free space 1356 generation 362173 owner 2789
leaf 301168066560 flags 0x1(WRITTEN) backref revision 1
fs uuid 922affe3-95d9-49e5-92aa-9fb3c4749279
chunk uuid 858737e2-d8ab-4e61-9ca3-95dc7035c5c5
        item 0 key (10550 DIR_ITEM 4272056) itemoff 16248 itemsize 35
                location key (8141473 INODE_ITEM 0) type FILE
                transid 270396 data_len 0 name_len 5
                name: kded4
        item 1 key (10550 DIR_ITEM 5045634) itemoff 16209 itemsize 39
                location key (7994499 INODE_ITEM 0) type FILE
                transid 270381 data_len 0 name_len 9
                name: buildhash
        item 2 key (10550 DIR_ITEM 6677674) itemoff 16171 itemsize 38
                location key (7939520 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 8
                name: sgitopnm
        item 3 key (10550 DIR_ITEM 7987325) itemoff 16137 itemsize 34
                location key (8531362 INODE_ITEM 0) type FILE
                transid 316669 data_len 0 name_len 4
                name: line
        item 4 key (10550 DIR_ITEM 8351974) itemoff 16100 itemsize 37
                location key (8024836 INODE_ITEM 0) type FILE
                transid 270385 data_len 0 name_len 7
                name: acyclic
        item 5 key (10550 DIR_ITEM 9045986) itemoff 16062 itemsize 38
                location key (8026789 INODE_ITEM 0) type FILE
                transid 270385 data_len 0 name_len 8
                name: isodebug
        item 6 key (10550 DIR_ITEM 9456603) itemoff 16024 itemsize 38
                location key (7939522 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 8
                name: sldtoppm
        item 7 key (10550 DIR_ITEM 10541543) itemoff 15983 itemsize 41
                location key (7899017 INODE_ITEM 0) type FILE
                transid 270374 data_len 0 name_len 11
                name: podofomerge
        item 8 key (10550 DIR_ITEM 15729738) itemoff 15940 itemsize 43
                location key (7899304 INODE_ITEM 0) type FILE
                transid 270375 data_len 0 name_len 13
                name: gsl-histogram
        item 9 key (10550 DIR_ITEM 16310152) itemoff 15897 itemsize 43
                location key (7893907 INODE_ITEM 0) type FILE
                transid 270372 data_len 0 name_len 13
                name: syslinux2ansi
        item 10 key (10550 DIR_ITEM 18458787) itemoff 15847 itemsize 50
                location key (8133501 INODE_ITEM 0) type FILE
                transid 270395 data_len 0 name_len 20
                name: xfce4-mouse-settings
        item 11 key (10550 DIR_ITEM 19291248) itemoff 15811 itemsize 36
                location key (7933583 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 6
                name: w3mman
        item 12 key (10550 DIR_ITEM 19507374) itemoff 15774 itemsize 37
                location key (7939391 INODE_ITEM 0) type SYMLINK
                transid 270378 data_len 0 name_len 7
                name: pnmcomp
        item 13 key (10550 DIR_ITEM 19513010) itemoff 15734 itemsize 40
                location key (8991250 INODE_ITEM 0) type FILE
                transid 360072 data_len 0 name_len 10
                name: pyvenv-3.6
        item 14 key (10550 DIR_ITEM 20904930) itemoff 15700 itemsize 34
                location key (7866544 INODE_ITEM 0) type FILE
                transid 270370 data_len 0 name_len 4
                name: znew
        item 15 key (10550 DIR_ITEM 21631592) itemoff 15662 itemsize 38
                location key (8123832 INODE_ITEM 0) type FILE
                transid 270394 data_len 0 name_len 8
                name: showfoto
        item 16 key (10550 DIR_ITEM 21694625) itemoff 15626 itemsize 36
                location key (8689298 INODE_ITEM 0) type FILE
                transid 323738 data_len 0 name_len 6
                name: smbtar
        item 17 key (10550 DIR_ITEM 22566358) itemoff 15593 itemsize 33
                location key (8122780 INODE_ITEM 0) type FILE
                transid 270394 data_len 0 name_len 3
                name: xsm
        item 18 key (10550 DIR_ITEM 23923035) itemoff 15555 itemsize 38
                location key (8025984 INODE_ITEM 0) type FILE
                transid 270385 data_len 0 name_len 8
                name: cdda2mp3
        item 19 key (10550 DIR_ITEM 25971396) itemoff 15521 itemsize 34
                location key (7914393 INODE_ITEM 0) type FILE
                transid 270376 data_len 0 name_len 4
                name: isql
        item 20 key (10550 DIR_ITEM 27240101) itemoff 15484 itemsize 37
                location key (7935759 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 7
                name: sg_zone
        item 21 key (10550 DIR_ITEM 29865559) itemoff 15450 itemsize 34
                location key (8921241 INODE_ITEM 0) type FILE
                transid 354217 data_len 0 name_len 4
                name: sudo
        item 22 key (10550 DIR_ITEM 30779597) itemoff 15414 itemsize 36
                location key (8973568 INODE_ITEM 0) type SYMLINK
                transid 357496 data_len 0 name_len 6
                name: stream
        item 23 key (10550 DIR_ITEM 31452410) itemoff 15377 itemsize 37
                location key (8869336 INODE_ITEM 0) type FILE
                transid 346840 data_len 0 name_len 7
                name: gpasswd
        item 24 key (10550 DIR_ITEM 31612432) itemoff 15342 itemsize 35
                location key (7895446 INODE_ITEM 0) type FILE
                transid 270373 data_len 0 name_len 5
                name: mailx
        item 25 key (10550 DIR_ITEM 34050395) itemoff 15304 itemsize 38
                location key (8906449 INODE_ITEM 0) type FILE
                transid 353566 data_len 0 name_len 8
                name: audacity
        item 26 key (10550 DIR_ITEM 34386064) itemoff 15258 itemsize 46
                location key (8575109 INODE_ITEM 0) type FILE
                transid 318251 data_len 0 name_len 16
                name: systemd-sysusers
        item 27 key (10550 DIR_ITEM 35160168) itemoff 15218 itemsize 40
                location key (7939337 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 10
                name: pbmtonokia
        item 28 key (10550 DIR_ITEM 35459757) itemoff 15166 itemsize 52
                location key (7898776 INODE_ITEM 0) type FILE
                transid 270374 data_len 0 name_len 22
                name: smp_conf_zone_perm_tbl
        item 29 key (10550 DIR_ITEM 39190276) itemoff 15125 itemsize 41
                location key (7899845 INODE_ITEM 0) type FILE
                transid 270375 data_len 0 name_len 11
                name: xsetpointer
        item 30 key (10550 DIR_ITEM 39339696) itemoff 15082 itemsize 43
                location key (7898783 INODE_ITEM 0) type FILE
                transid 270374 data_len 0 name_len 13
                name: smp_read_gpio
        item 31 key (10550 DIR_ITEM 47322961) itemoff 15035 itemsize 47
                location key (8136020 INODE_ITEM 0) type FILE
                transid 270395 data_len 0 name_len 17
                name: xfce4-popup-notes
        item 32 key (10550 DIR_ITEM 49690564) itemoff 14996 itemsize 39
                location key (8120353 INODE_ITEM 0) type FILE
                transid 270394 data_len 0 name_len 9
                name: snmptable
        item 33 key (10550 DIR_ITEM 51907865) itemoff 14960 itemsize 36
                location key (7981635 INODE_ITEM 0) type FILE
                transid 270380 data_len 0 name_len 6
                name: qmicli
        item 34 key (10550 DIR_ITEM 54445504) itemoff 14917 itemsize 43
                location key (7898798 INODE_ITEM 0) type FILE
                transid 270374 data_len 0 name_len 13
                name: smp_zone_lock
        item 35 key (10550 DIR_ITEM 56507565) itemoff 14876 itemsize 41
                location key (8284522 INODE_ITEM 0) type FILE
                transid 283299 data_len 0 name_len 11
                name: strip-id3v1
        item 36 key (10550 DIR_ITEM 57221819) itemoff 14833 itemsize 43
                location key (7899019 INODE_ITEM 0) type FILE
                transid 270374 data_len 0 name_len 13
                name: podofopdfinfo
        item 37 key (10550 DIR_ITEM 60433716) itemoff 14785 itemsize 48
                location key (7898780 INODE_ITEM 0) type FILE
                transid 270374 data_len 0 name_len 18
                name: smp_ena_dis_zoning
        item 38 key (10550 DIR_ITEM 61266890) itemoff 14749 itemsize 36
                location key (7935761 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 6
                name: sgm_dd
        item 39 key (10550 DIR_ITEM 65063325) itemoff 14711 itemsize 38
                location key (7981262 INODE_ITEM 0) type FILE
                transid 270380 data_len 0 name_len 8
                name: smixlate
        item 40 key (10550 DIR_ITEM 65415951) itemoff 14669 itemsize 42
                location key (7866026 INODE_ITEM 0) type FILE
                transid 270369 data_len 0 name_len 12
                name: amuFormat.sh
        item 41 key (10550 DIR_ITEM 65704522) itemoff 14632 itemsize 37
                location key (7939398 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 7
                name: pnmflip
        item 42 key (10550 DIR_ITEM 67853505) itemoff 14576 itemsize 56
                location key (7928159 INODE_ITEM 0) type FILE
                transid 270377 data_len 0 name_len 26
                name: exo-preferred-applications
        item 43 key (10550 DIR_ITEM 67922953) itemoff 14538 itemsize 38
                location key (7939344 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 8
                name: pbmtoptx
        item 44 key (10550 DIR_ITEM 68847419) itemoff 14505 itemsize 33
                location key (7854494 INODE_ITEM 0) type FILE
                transid 270369 data_len 0 name_len 3
                name: env
        item 45 key (10550 DIR_ITEM 69918749) itemoff 14469 itemsize 36
                location key (7897856 INODE_ITEM 0) type SYMLINK
                transid 270374 data_len 0 name_len 6
                name: bzless
        item 46 key (10550 DIR_ITEM 72137471) itemoff 14434 itemsize 35
                location key (8511818 INODE_ITEM 0) type FILE
                transid 313738 data_len 0 name_len 5
                name: iconv
        item 47 key (10550 DIR_ITEM 72316328) itemoff 14391 itemsize 43
                location key (8028853 INODE_ITEM 0) type FILE
                transid 270385 data_len 0 name_len 13
                name: kbuildsycoca5
        item 48 key (10550 DIR_ITEM 73595871) itemoff 14340 itemsize 51
                location key (8100256 INODE_ITEM 0) type FILE
                transid 270392 data_len 0 name_len 21
                name: qtattributionsscanner
        item 49 key (10550 DIR_ITEM 74618706) itemoff 14298 itemsize 42
                location key (7996184 INODE_ITEM 0) type FILE
                transid 270382 data_len 0 name_len 12
                name: hugin_lensdb
        item 50 key (10550 DIR_ITEM 77115367) itemoff 14259 itemsize 39
                location key (7899007 INODE_ITEM 0) type FILE
                transid 270374 data_len 0 name_len 9
                name: podofobox
        item 51 key (10550 DIR_ITEM 79947135) itemoff 14213 itemsize 46
                location key (7899009 INODE_ITEM 0) type FILE
                transid 270374 data_len 0 name_len 16
                name: podofocountpages
        item 52 key (10550 DIR_ITEM 80173633) itemoff 14165 itemsize 48
                location key (7979109 INODE_ITEM 0) type FILE
                transid 270380 data_len 0 name_len 18
                name: clean-binary-files
        item 53 key (10550 DIR_ITEM 80250676) itemoff 14129 itemsize 36
                location key (7931259 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 6
                name: ypchfn
        item 54 key (10550 DIR_ITEM 80414892) itemoff 14086 itemsize 43
                location key (8575103 INODE_ITEM 0) type FILE
                transid 318251 data_len 0 name_len 13
                name: systemd-mount
        item 55 key (10550 DIR_ITEM 80434688) itemoff 14043 itemsize 43
                location key (7988498 INODE_ITEM 0) type FILE
                transid 270381 data_len 0 name_len 13
                name: automake-1.15
        item 56 key (10550 DIR_ITEM 81220010) itemoff 14001 itemsize 42
                location key (7942464 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 12
                name: lessclose.sh
        item 57 key (10550 DIR_ITEM 81223716) itemoff 13963 itemsize 38
                location key (7939372 INODE_ITEM 0) type SYMLINK
                transid 270378 data_len 0 name_len 8
                name: pgmslice
        item 58 key (10550 DIR_ITEM 83569019) itemoff 13921 itemsize 42
                location key (8096432 INODE_ITEM 0) type FILE
                transid 270391 data_len 0 name_len 12
                name: linguist-qt5
        item 59 key (10550 DIR_ITEM 85923924) itemoff 13882 itemsize 39
                location key (7940592 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 9
                name: sctp_test
        item 60 key (10550 DIR_ITEM 86107397) itemoff 13846 itemsize 36
                location key (8219548 INODE_ITEM 0) type FILE
                transid 276588 data_len 0 name_len 6
                name: brltty
        item 61 key (10550 DIR_ITEM 86131131) itemoff 13808 itemsize 38
                location key (7939331 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 8
                name: pbmtolps
        item 62 key (10550 DIR_ITEM 88769685) itemoff 13770 itemsize 38
                location key (8689289 INODE_ITEM 0) type FILE
                transid 323738 data_len 0 name_len 8
                name: sharesec
        item 63 key (10550 DIR_ITEM 89584778) itemoff 13726 itemsize 44
                location key (8575087 INODE_ITEM 0) type FILE
                transid 318251 data_len 0 name_len 14
                name: kernel-install
        item 64 key (10550 DIR_ITEM 92596335) itemoff 13687 itemsize 39
                location key (7900484 INODE_ITEM 0) type FILE
                transid 270375 data_len 0 name_len 9
                name: pdfdetach
        item 65 key (10550 DIR_ITEM 93619923) itemoff 13652 itemsize 35
                location key (7893844 INODE_ITEM 0) type FILE
                transid 270372 data_len 0 name_len 5
                name: whois
        item 66 key (10550 DIR_ITEM 94198366) itemoff 13617 itemsize 35
                location key (7933490 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 5
                name: xclip
        item 67 key (10550 DIR_ITEM 95455750) itemoff 13581 itemsize 36
                location key (8105594 INODE_ITEM 0) type FILE
                transid 270392 data_len 0 name_len 6
                name: knotes
        item 68 key (10550 DIR_ITEM 96561063) itemoff 13548 itemsize 33
                location key (8823315 INODE_ITEM 0) type FILE
                transid 341614 data_len 0 name_len 3
                name: tor
        item 69 key (10550 DIR_ITEM 99974273) itemoff 13503 itemsize 45
                location key (8575101 INODE_ITEM 0) type FILE
                transid 318251 data_len 0 name_len 15
                name: systemd-inhibit
        item 70 key (10550 DIR_ITEM 100619323) itemoff 13466 itemsize 37
                location key (8531363 INODE_ITEM 0) type SYMLINK
                transid 316669 data_len 0 name_len 7
                name: linux32
        item 71 key (10550 DIR_ITEM 101879870) itemoff 13416 itemsize 50
                location key (7933478 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 20
                name: xdg-user-dirs-update
        item 72 key (10550 DIR_ITEM 101904877) itemoff 13379 itemsize 37
                location key (7901334 INODE_ITEM 0) type FILE
                transid 270375 data_len 0 name_len 7
                name: editres
        item 73 key (10550 DIR_ITEM 103442851) itemoff 13338 itemsize 41
                location key (7939390 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 11
                name: pnmcolormap
        item 74 key (10550 DIR_ITEM 103997408) itemoff 13297 itemsize 41
                location key (7884516 INODE_ITEM 0) type FILE
                transid 270371 data_len 0 name_len 11
                name: dbus-daemon
        item 75 key (10550 DIR_ITEM 108382495) itemoff 13225 itemsize 72
                location key (74333 INODE_ITEM 0) type FILE
                transid 58 data_len 0 name_len 6
                name: newg
                location key (8869339 INODE_ITEM 0) type FILE
                transid 346840 data_len 0 name_len 6
                name: newgrp
        item 76 key (10550 DIR_ITEM 111480647) itemoff 13182 itemsize 43
                location key (7881436 INODE_ITEM 0) type FILE
                transid 270371 data_len 0 name_len 13
                name: unicode_start
        item 77 key (10550 DIR_ITEM 112154537) itemoff 13142 itemsize 40
                location key (8644116 INODE_ITEM 0) type SYMLINK
                transid 321577 data_len 0 name_len 10
                name: policytool
        item 78 key (10550 DIR_ITEM 118620247) itemoff 13098 itemsize 44
                location key (8813716 INODE_ITEM 0) type FILE
                transid 339881 data_len 0 name_len 14
                name: qemu-pr-helper
        item 79 key (10550 DIR_ITEM 118987452) itemoff 13064 itemsize 34
                location key (7876009 INODE_ITEM 0) type FILE
                transid 270370 data_len 0 name_len 4
                name: lsof
        item 80 key (10550 DIR_ITEM 119569851) itemoff 13026 itemsize 38
                location key (8427981 INODE_ITEM 0) type FILE
                transid 305712 data_len 0 name_len 8
                name: pod2text
        item 81 key (10550 DIR_ITEM 119705705) itemoff 12982 itemsize 44
                location key (8097732 INODE_ITEM 0) type FILE
                transid 270392 data_len 0 name_len 14
                name: keditbookmarks
        item 82 key (10550 DIR_ITEM 121112906) itemoff 12942 itemsize 40
                location key (8746998 INODE_ITEM 0) type FILE
                transid 332903 data_len 0 name_len 10
                name: gvfs-trash
        item 83 key (10550 DIR_ITEM 125975686) itemoff 12905 itemsize 37
                location key (8032508 INODE_ITEM 0) type FILE
                transid 270385 data_len 0 name_len 7
                name: kshell5
        item 84 key (10550 DIR_ITEM 125990854) itemoff 12855 itemsize 50
                location key (8048286 INODE_ITEM 0) type FILE
                transid 270386 data_len 0 name_len 20
                name: tst_inactivity_timer
        item 85 key (10550 DIR_ITEM 126732258) itemoff 12813 itemsize 42
                location key (7939504 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 12
                name: ppmtowinicon
        item 86 key (10550 DIR_ITEM 131798857) itemoff 12776 itemsize 37
                location key (7939360 INODE_ITEM 0) type SYMLINK
                transid 270378 data_len 0 name_len 7
                name: pgmedge
        item 87 key (10550 DIR_ITEM 135945193) itemoff 12738 itemsize 38
                location key (7939210 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 8
                name: imgtoppm
        item 88 key (10550 DIR_ITEM 139657832) itemoff 12702 itemsize 36
                location key (8628921 INODE_ITEM 0) type FILE
                transid 320229 data_len 0 name_len 6
                name: ssltap
        item 89 key (10550 DIR_ITEM 140165687) itemoff 12666 itemsize 36
                location key (8924918 INODE_ITEM 0) type SYMLINK
                transid 354641 data_len 0 name_len 6
                name: python
        item 90 key (10550 DIR_ITEM 140452253) itemoff 12629 itemsize 37
                location key (8883295 INODE_ITEM 0) type FILE
                transid 349443 data_len 0 name_len 7
                name: gdtopng
        item 91 key (10550 DIR_ITEM 149453168) itemoff 12586 itemsize 43
                location key (7960876 INODE_ITEM 0) type SYMLINK
                transid 270380 data_len 0 name_len 13
                name: db_checkpoint
        item 92 key (10550 DIR_ITEM 149489412) itemoff 12541 itemsize 45
                location key (7899434 INODE_ITEM 0) type FILE
                transid 270375 data_len 0 name_len 15
                name: .fipscheck.hmac
        item 93 key (10550 DIR_ITEM 153074219) itemoff 12500 itemsize 41
                location key (7899415 INODE_ITEM 0) type FILE
                transid 270375 data_len 0 name_len 11
                name: mkfontscale
        item 94 key (10550 DIR_ITEM 153200413) itemoff 12465 itemsize 35
                location key (7897815 INODE_ITEM 0) type FILE
                transid 270374 data_len 0 name_len 5
                name: dconf
        item 95 key (10550 DIR_ITEM 153853859) itemoff 12426 itemsize 39
                location key (7987028 INODE_ITEM 0) type FILE
                transid 270381 data_len 0 name_len 9
                name: xf86debug
        item 96 key (10550 DIR_ITEM 154386569) itemoff 12383 itemsize 43
                location key (7881434 INODE_ITEM 0) type FILE
                transid 270371 data_len 0 name_len 13
                name: spawn_console
        item 97 key (10550 DIR_ITEM 155341665) itemoff 12345 itemsize 38
                location key (7935704 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 8
                name: sg_ident
        item 98 key (10550 DIR_ITEM 156246632) itemoff 12297 itemsize 48
                location key (9013755 INODE_ITEM 0) type FILE
                transid 362173 data_len 0 name_len 18
                name: mysqld_safe_helper
        item 99 key (10550 DIR_ITEM 156556898) itemoff 12251 itemsize 46
                location key (7989793 INODE_ITEM 0) type FILE
                transid 270381 data_len 0 name_len 16
                name: ntfs-3g.secaudit
        item 100 key (10550 DIR_ITEM 156960508) itemoff 12209 itemsize 42
                location key (8026570 INODE_ITEM 0) type FILE
                transid 270385 data_len 0 name_len 12
                name: quadrapassel
        item 101 key (10550 DIR_ITEM 157186081) itemoff 12167 itemsize 42
                location key (8893710 INODE_ITEM 0) type SYMLINK
                transid 351304 data_len 0 name_len 12
                name: ebook-device
        item 102 key (10550 DIR_ITEM 157582155) itemoff 12132 itemsize 35
                location key (7901120 INODE_ITEM 0) type FILE
                transid 270375 data_len 0 name_len 5
                name: luitx
        item 103 key (10550 DIR_ITEM 160449584) itemoff 12097 itemsize 35
                location key (7866006 INODE_ITEM 0) type FILE
                transid 270369 data_len 0 name_len 5
                name: clear
        item 104 key (10550 DIR_ITEM 160460297) itemoff 12062 itemsize 35
                location key (7989301 INODE_ITEM 0) type FILE
                transid 270381 data_len 0 name_len 5
                name: f2py2
        item 105 key (10550 DIR_ITEM 160878447) itemoff 12024 itemsize 38
                location key (7939325 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 8
                name: pbmtogem
        item 106 key (10550 DIR_ITEM 162153544) itemoff 11978 itemsize 46
                location key (7899014 INODE_ITEM 0) type FILE
                transid 270374 data_len 0 name_len 16
                name: podofoimgextract
        item 107 key (10550 DIR_ITEM 165993558) itemoff 11942 itemsize 36
                location key (8968899 INODE_ITEM 0) type FILE
                transid 356706 data_len 0 name_len 6
                name: cheese
        item 108 key (10550 DIR_ITEM 166426468) itemoff 11902 itemsize 40
                location key (8892798 INODE_ITEM 0) type FILE
                transid 351049 data_len 0 name_len 10
                name: gnutls-cli
        item 109 key (10550 DIR_ITEM 166561401) itemoff 11865 itemsize 37
                location key (8530883 INODE_ITEM 0) type FILE
                transid 316669 data_len 0 name_len 7
                name: pgpring
        item 110 key (10550 DIR_ITEM 168408717) itemoff 11827 itemsize 38
                location key (8098091 INODE_ITEM 0) type FILE
                transid 270392 data_len 0 name_len 8
                name: balooctl
        item 111 key (10550 DIR_ITEM 170171290) itemoff 11784 itemsize 43
                location key (7977531 INODE_ITEM 0) type FILE
                transid 270380 data_len 0 name_len 13
                name: acpi_validate
        item 112 key (10550 DIR_ITEM 171788808) itemoff 11745 itemsize 39
                location key (8016247 INODE_ITEM 0) type FILE
                transid 270384 data_len 0 name_len 9
                name: checkXML5
        item 113 key (10550 DIR_ITEM 172309332) itemoff 11712 itemsize 33
                location key (7897569 INODE_ITEM 0) type FILE
                transid 270374 data_len 0 name_len 3
                name: ico
        item 114 key (10550 DIR_ITEM 172351642) itemoff 11673 itemsize 39
                location key (7960144 INODE_ITEM 0) type FILE
                transid 270380 data_len 0 name_len 9
                name: dislocate
        item 115 key (10550 DIR_ITEM 172376766) itemoff 11633 itemsize 40
                location key (7940492 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 10
                name: yuvdenoise
        item 116 key (10550 DIR_ITEM 174662705) itemoff 11596 itemsize 37
                location key (7894055 INODE_ITEM 0) type FILE
                transid 270373 data_len 0 name_len 7
                name: showrgb
        item 117 key (10550 DIR_ITEM 175083948) itemoff 11555 itemsize 41
                location key (7987206 INODE_ITEM 0) type FILE
                transid 270381 data_len 0 name_len 11
                name: getfilename
        item 118 key (10550 DIR_ITEM 175938737) itemoff 11520 itemsize 35
                location key (8531368 INODE_ITEM 0) type FILE
                transid 316669 data_len 0 name_len 5
                name: lsipc
        item 119 key (10550 DIR_ITEM 175956747) itemoff 11483 itemsize 37
                location key (8732426 INODE_ITEM 0) type FILE
                transid 330534 data_len 0 name_len 7
                name: ooffice
        item 120 key (10550 DIR_ITEM 176051772) itemoff 11450 itemsize 33
                location key (7854562 INODE_ITEM 0) type FILE
                transid 270369 data_len 0 name_len 3
                name: tty
        item 121 key (10550 DIR_ITEM 176233385) itemoff 11414 itemsize 36
                location key (7928999 INODE_ITEM 0) type FILE
                transid 270377 data_len 0 name_len 6
                name: cdrdao
        item 122 key (10550 DIR_ITEM 177601502) itemoff 11373 itemsize 41
                location key (8120361 INODE_ITEM 0) type FILE
                transid 270394 data_len 0 name_len 11
                name: traptoemail
        item 123 key (10550 DIR_ITEM 177790842) itemoff 11339 itemsize 34
                location key (7881401 INODE_ITEM 0) type FILE
                transid 270371 data_len 0 name_len 4
                name: chvt
        item 124 key (10550 DIR_ITEM 178503331) itemoff 11305 itemsize 34
                location key (7901177 INODE_ITEM 0) type FILE
                transid 270375 data_len 0 name_len 4
                name: xmag
        item 125 key (10550 DIR_ITEM 178633898) itemoff 11273 itemsize 32
                location key (7854516 INODE_ITEM 0) type FILE
                transid 270369 data_len 0 name_len 2
                name: mv
        item 126 key (10550 DIR_ITEM 179121774) itemoff 11239 itemsize 34
                location key (8869334 INODE_ITEM 0) type FILE
                transid 346840 data_len 0 name_len 4
                name: chsh
        item 127 key (10550 DIR_ITEM 180689377) itemoff 11204 itemsize 35
                location key (7854476 INODE_ITEM 0) type FILE
                transid 270369 data_len 0 name_len 5
                name: chcon
        item 128 key (10550 DIR_ITEM 182051214) itemoff 11166 itemsize 38
                location key (8220173 INODE_ITEM 0) type FILE
                transid 276588 data_len 0 name_len 8
                name: vm-disks
        item 129 key (10550 DIR_ITEM 182474039) itemoff 11128 itemsize 38
                location key (8219114 INODE_ITEM 0) type FILE
                transid 276588 data_len 0 name_len 8
                name: zerofree
        item 130 key (10550 DIR_ITEM 185453414) itemoff 11087 itemsize 41
                location key (8097450 INODE_ITEM 0) type FILE
                transid 270391 data_len 0 name_len 11
                name: kwatchgnupg
        item 131 key (10550 DIR_ITEM 186561017) itemoff 11032 itemsize 55
                location key (8133984 INODE_ITEM 0) type FILE
                transid 270395 data_len 0 name_len 25
                name: xfce4-popup-directorymenu
        item 132 key (10550 DIR_ITEM 187400506) itemoff 10969 itemsize 63
                location key (8098060 INODE_ITEM 0) type FILE
                transid 270392 data_len 0 name_len 33
                name: baloo_filemetadata_temp_extractor
        item 133 key (10550 DIR_ITEM 188435761) itemoff 10926 itemsize 43
                location key (7887199 INODE_ITEM 0) type FILE
                transid 270371 data_len 0 name_len 13
                name: nosetests-3.6
        item 134 key (10550 DIR_ITEM 190450965) itemoff 10891 itemsize 35
                location key (7900030 INODE_ITEM 0) type FILE
                transid 270375 data_len 0 name_len 5
                name: sshfs
        item 135 key (10550 DIR_ITEM 190532504) itemoff 10843 itemsize 48
                location key (8133470 INODE_ITEM 0) type FILE
                transid 270395 data_len 0 name_len 18
                name: xfdesktop-settings
        item 136 key (10550 DIR_ITEM 191076901) itemoff 10806 itemsize 37
                location key (8689285 INODE_ITEM 0) type FILE
                transid 323738 data_len 0 name_len 7
                name: regtree
        item 137 key (10550 DIR_ITEM 191435277) itemoff 10770 itemsize 36
                location key (8976684 INODE_ITEM 0) type FILE
                transid 357862 data_len 0 name_len 6
                name: ffmpeg
        item 138 key (10550 DIR_ITEM 191775916) itemoff 10732 itemsize 38
                location key (8097112 INODE_ITEM 0) type FILE
                transid 270391 data_len 0 name_len 8
                name: kreversi
        item 139 key (10550 DIR_ITEM 193207812) itemoff 10695 itemsize 37
                location key (7989748 INODE_ITEM 0) type FILE
                transid 270381 data_len 0 name_len 7
                name: ntfscat
        item 140 key (10550 DIR_ITEM 196767418) itemoff 10657 itemsize 38
                location key (7938889 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 8
                name: fixpspps
        item 141 key (10550 DIR_ITEM 197021103) itemoff 10609 itemsize 48
                location key (8125473 INODE_ITEM 0) type FILE
                transid 270395 data_len 0 name_len 18
                name: policyeditor.itweb
        item 142 key (10550 DIR_ITEM 199192314) itemoff 10568 itemsize 41
                location key (8036764 INODE_ITEM 0) type FILE
                transid 270386 data_len 0 name_len 11
                name: akonaditest
        item 143 key (10550 DIR_ITEM 200350540) itemoff 10530 itemsize 38
                location key (7895486 INODE_ITEM 0) type FILE
                transid 270373 data_len 0 name_len 8
                name: xsltproc
        item 144 key (10550 DIR_ITEM 200855103) itemoff 10491 itemsize 39
                location key (8133469 INODE_ITEM 0) type FILE
                transid 270395 data_len 0 name_len 9
                name: xfdesktop
        item 145 key (10550 DIR_ITEM 201064207) itemoff 10451 itemsize 40
                location key (7893767 INODE_ITEM 0) type SYMLINK
                transid 270372 data_len 0 name_len 10
                name: xloadimage
        item 146 key (10550 DIR_ITEM 202847040) itemoff 10411 itemsize 40
                location key (8828948 INODE_ITEM 0) type FILE
                transid 342359 data_len 0 name_len 10
                name: comps2solv
        item 147 key (10550 DIR_ITEM 205151558) itemoff 10370 itemsize 41
                location key (7939444 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 11
                name: ppmbrighten
        item 148 key (10550 DIR_ITEM 206822491) itemoff 10328 itemsize 42
                location key (8024595 INODE_ITEM 0) type SYMLINK
                transid 270384 data_len 0 name_len 12
                name: transmission
        item 149 key (10550 DIR_ITEM 208645618) itemoff 10291 itemsize 37
                location key (7939452 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 7
                name: ppmdist
        item 150 key (10550 DIR_ITEM 209237997) itemoff 10233 itemsize 58
                location key (8123824 INODE_ITEM 0) type FILE
                transid 270394 data_len 0 name_len 28
                name: system-config-printer-applet
        item 151 key (10550 DIR_ITEM 209787167) itemoff 10190 itemsize 43
                location key (7939269 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 13
                name: pamstereogram
        item 152 key (10550 DIR_ITEM 210137934) itemoff 10139 itemsize 51
                location key (8038463 INODE_ITEM 0) type FILE
                transid 270386 data_len 0 name_len 21
                name: akonadi_mbox_resource
        item 153 key (10550 DIR_ITEM 210445819) itemoff 10095 itemsize 44
                location key (7996182 INODE_ITEM 0) type FILE
                transid 270382 data_len 0 name_len 14
                name: hugin_executor
        item 154 key (10550 DIR_ITEM 212942227) itemoff 10054 itemsize 41
                location key (8120348 INODE_ITEM 0) type FILE
                transid 270394 data_len 0 name_len 11
                name: snmpgetnext
        item 155 key (10550 DIR_ITEM 213367364) itemoff 10015 itemsize 39
                location key (7939217 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 9
                name: macptopbm
        item 156 key (10550 DIR_ITEM 214922785) itemoff 9981 itemsize 34
                location key (7899149 INODE_ITEM 0) type FILE
                transid 270375 data_len 0 name_len 4
                name: lame
        item 157 key (10550 DIR_ITEM 218725611) itemoff 9943 itemsize 38
                location key (7899950 INODE_ITEM 0) type FILE
                transid 270375 data_len 0 name_len 8
                name: showfont
        item 158 key (10550 DIR_ITEM 219296885) itemoff 9910 itemsize 33
                location key (7866031 INODE_ITEM 0) type SYMLINK
                transid 270369 data_len 0 name_len 3
                name: mcd
        item 159 key (10550 DIR_ITEM 219477337) itemoff 9875 itemsize 35
                location key (7987044 INODE_ITEM 0) type SYMLINK
                transid 270381 data_len 0 name_len 5
                name: pkill
        item 160 key (10550 DIR_ITEM 220690167) itemoff 9828 itemsize 47
                location key (7978163 INODE_ITEM 0) type FILE
                transid 270380 data_len 0 name_len 17
                name: rdoc.ruby.ruby2.5
        item 161 key (10550 DIR_ITEM 223365983) itemoff 9790 itemsize 38
                location key (7899122 INODE_ITEM 0) type FILE
                transid 270375 data_len 0 name_len 8
                name: numademo
        item 162 key (10550 DIR_ITEM 223702966) itemoff 9754 itemsize 36
                location key (7865908 INODE_ITEM 0) type SYMLINK
                transid 270369 data_len 0 name_len 6
                name: lzless
        item 163 key (10550 DIR_ITEM 224567181) itemoff 9712 itemsize 42
                location key (7928752 INODE_ITEM 0) type FILE
                transid 270377 data_len 0 name_len 12
                name: pax11publish
        item 164 key (10550 DIR_ITEM 224821798) itemoff 9672 itemsize 40
                location key (7866053 INODE_ITEM 0) type SYMLINK
                transid 270369 data_len 0 name_len 10
                name: mtoolstest
        item 165 key (10550 DIR_ITEM 227087995) itemoff 9633 itemsize 39
                location key (8828952 INODE_ITEM 0) type FILE
                transid 342359 data_len 0 name_len 9
                name: mergesolv
        item 166 key (10550 DIR_ITEM 227676689) itemoff 9598 itemsize 35
                location key (7868662 INODE_ITEM 0) type FILE
                transid 270370 data_len 0 name_len 5
                name: eu-nm
        item 167 key (10550 DIR_ITEM 228577476) itemoff 9556 itemsize 42
                location key (7881417 INODE_ITEM 0) type FILE
                transid 270371 data_len 0 name_len 12
                name: outpsfheader
        item 168 key (10550 DIR_ITEM 230832956) itemoff 9519 itemsize 37
                location key (7881412 INODE_ITEM 0) type FILE
                transid 270371 data_len 0 name_len 7
                name: kbdrate
        item 169 key (10550 DIR_ITEM 231268802) itemoff 9477 itemsize 42
                location key (7940473 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 12
                name: y4mcolorbars
        item 170 key (10550 DIR_ITEM 232112477) itemoff 9433 itemsize 44
                location key (7994076 INODE_ITEM 0) type FILE
                transid 270381 data_len 0 name_len 14
                name: ibus-autostart
        item 171 key (10550 DIR_ITEM 234485755) itemoff 9395 itemsize 38
                location key (7939490 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 8
                name: ppmtopcx
        item 172 key (10550 DIR_ITEM 237565348) itemoff 9351 itemsize 44
                location key (8575112 INODE_ITEM 0) type SYMLINK
                transid 318251 data_len 0 name_len 14
                name: systemd-umount
        item 173 key (10550 DIR_ITEM 238678335) itemoff 9313 itemsize 38
                location key (7931259 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 8
                name: yppasswd
        item 174 key (10550 DIR_ITEM 242698714) itemoff 9274 itemsize 39
                location key (8024867 INODE_ITEM 0) type SYMLINK
                transid 270385 data_len 0 name_len 9
                name: patchwork
        item 175 key (10550 DIR_ITEM 243820105) itemoff 9237 itemsize 37
                location key (7940466 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 7
                name: png2yuv
        item 176 key (10550 DIR_ITEM 244921432) itemoff 9199 itemsize 38
                location key (8120357 INODE_ITEM 0) type FILE
                transid 270394 data_len 0 name_len 8
                name: snmptrap
        item 177 key (10550 DIR_ITEM 245084423) itemoff 9164 itemsize 35
                location key (7854479 INODE_ITEM 0) type FILE
                transid 270369 data_len 0 name_len 5
                name: chown
        item 178 key (10550 DIR_ITEM 245661001) itemoff 9128 itemsize 36
                location key (8123421 INODE_ITEM 0) type FILE
                transid 270394 data_len 0 name_len 6
                name: upower
        item 179 key (10550 DIR_ITEM 246787026) itemoff 9086 itemsize 42
                location key (8828954 INODE_ITEM 0) type SYMLINK
                transid 342359 data_len 0 name_len 12
                name: repo2solv.sh
        item 180 key (10550 DIR_ITEM 249561580) itemoff 9046 itemsize 40
                location key (7939301 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 10
                name: pamwipeout
        item 181 key (10550 DIR_ITEM 251371264) itemoff 8990 itemsize 56
                location key (8123895 INODE_ITEM 0) type FILE
                transid 270394 data_len 0 name_len 26
                name: ipod-read-sysinfo-extended
        item 182 key (10550 DIR_ITEM 253147454) itemoff 8952 itemsize 38
                location key (7854527 INODE_ITEM 0) type FILE
                transid 270369 data_len 0 name_len 8
                name: printenv
        item 183 key (10550 DIR_ITEM 253515254) itemoff 8912 itemsize 40
                location key (7938896 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 10
                name: includeres
        item 184 key (10550 DIR_ITEM 254923011) itemoff 8879 itemsize 33
                location key (7854530 INODE_ITEM 0) type FILE
                transid 270369 data_len 0 name_len 3
                name: pwd
        item 185 key (10550 DIR_ITEM 255183178) itemoff 8844 itemsize 35
                location key (7928747 INODE_ITEM 0) type SYMLINK
                transid 270377 data_len 0 name_len 5
                name: pamon
        item 186 key (10550 DIR_ITEM 255437966) itemoff 8804 itemsize 40
                location key (9013753 INODE_ITEM 0) type FILE
                transid 362173 data_len 0 name_len 10
                name: mysqladmin
        item 187 key (10550 DIR_ITEM 259279386) itemoff 8766 itemsize 38
                location key (7894923 INODE_ITEM 0) type FILE
                transid 270373 data_len 0 name_len 8
                name: socklist
        item 188 key (10550 DIR_ITEM 259927578) itemoff 8719 itemsize 47
                location key (8885574 INODE_ITEM 0) type FILE
                transid 349820 data_len 0 name_len 17
                name: grub2-menulst2cfg
        item 189 key (10550 DIR_ITEM 263451273) itemoff 8685 itemsize 34
                location key (7866036 INODE_ITEM 0) type SYMLINK
                transid 270369 data_len 0 name_len 4
                name: mdel
        item 190 key (10550 DIR_ITEM 263682508) itemoff 8648 itemsize 37
                location key (7938894 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 7
                name: fixwwps
        item 191 key (10550 DIR_ITEM 264487110) itemoff 8611 itemsize 37
                location key (8531387 INODE_ITEM 0) type FILE
                transid 316669 data_len 0 name_len 7
                name: setpriv
        item 192 key (10550 DIR_ITEM 264713521) itemoff 8577 itemsize 34
                location key (7940429 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 4
                name: lksh
        item 193 key (10550 DIR_ITEM 266102022) itemoff 8539 itemsize 38
                location key (7939481 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 8
                name: ppmtogif
        item 194 key (10550 DIR_ITEM 267985614) itemoff 8496 itemsize 43
                location key (8145058 INODE_ITEM 0) type FILE
                transid 270396 data_len 0 name_len 13
                name: keditfiletype
        item 195 key (10550 DIR_ITEM 268007155) itemoff 8458 itemsize 38
                location key (7940468 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 8
                name: ppmtoy4m
        item 196 key (10550 DIR_ITEM 269116135) itemoff 8424 itemsize 34
                location key (8145054 INODE_ITEM 0) type FILE
                transid 270396 data_len 0 name_len 4
                name: kde4
        item 197 key (10550 DIR_ITEM 271305115) itemoff 8379 itemsize 45
                location key (7935697 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 15
                name: sg_copy_results
        item 198 key (10550 DIR_ITEM 273021970) itemoff 8342 itemsize 37
                location key (8425877 INODE_ITEM 0) type FILE
                transid 305711 data_len 0 name_len 7
                name: hcidump
        item 199 key (10550 DIR_ITEM 274012901) itemoff 8305 itemsize 37
                location key (7960878 INODE_ITEM 0) type SYMLINK
                transid 270380 data_len 0 name_len 7
                name: db_dump
        item 200 key (10550 DIR_ITEM 275396121) itemoff 8258 itemsize 47
                location key (8141475 INODE_ITEM 0) type FILE
                transid 270396 data_len 0 name_len 17
                name: kdeinit4_shutdown
        item 201 key (10550 DIR_ITEM 276002597) itemoff 8223 itemsize 35
                location key (8012546 INODE_ITEM 0) type FILE
                transid 270383 data_len 0 name_len 5
                name: munch
        item 202 key (10550 DIR_ITEM 280303165) itemoff 8184 itemsize 39
                location key (8878476 INODE_ITEM 0) type FILE
                transid 348777 data_len 0 name_len 9
                name: icewmtray
        item 203 key (10550 DIR_ITEM 281052186) itemoff 8147 itemsize 37
                location key (8531379 INODE_ITEM 0) type FILE
                transid 316669 data_len 0 name_len 7
                name: nsenter
        item 204 key (10550 DIR_ITEM 281137205) itemoff 8109 itemsize 38
                location key (7897892 INODE_ITEM 0) type FILE
                transid 270374 data_len 0 name_len 8
                name: getfattr
        item 205 key (10550 DIR_ITEM 283498211) itemoff 8076 itemsize 33
                location key (7978683 INODE_ITEM 0) type FILE
                transid 270380 data_len 0 name_len 3
                name: pic
        item 206 key (10550 DIR_ITEM 284394741) itemoff 8039 itemsize 37
                location key (7987980 INODE_ITEM 0) type FILE
                transid 270381 data_len 0 name_len 7
                name: pfbtops
        item 207 key (10550 DIR_ITEM 287254989) itemoff 8001 itemsize 38
                location key (7987885 INODE_ITEM 0) type FILE
                transid 270381 data_len 0 name_len 8
                name: aseqdump
        item 208 key (10550 DIR_ITEM 287622286) itemoff 7963 itemsize 38
                location key (7939221 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 8
                name: mrftopbm
        item 209 key (10550 DIR_ITEM 288289702) itemoff 7915 itemsize 48
                location key (7939333 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 18
                name: pbmtomatrixorbital
        item 210 key (10550 DIR_ITEM 289396669) itemoff 7879 itemsize 36
                location key (7976671 INODE_ITEM 0) type FILE
                transid 270380 data_len 0 name_len 6
                name: ld.bfd
        item 211 key (10550 DIR_ITEM 290169847) itemoff 7827 itemsize 52
                location key (9006138 INODE_ITEM 0) type FILE
                transid 361523 data_len 0 name_len 22
                name: nvidia-cuda-mps-server
        item 212 key (10550 DIR_ITEM 291206685) itemoff 7780 itemsize 47
                location key (8100727 INODE_ITEM 0) type FILE
                transid 270392 data_len 0 name_len 17
                name: gwenview_importer
        item 213 key (10550 DIR_ITEM 291944630) itemoff 7744 itemsize 36
                location key (7899796 INODE_ITEM 0) type FILE
                transid 270375 data_len 0 name_len 6
                name: xvinfo
        item 214 key (10550 DIR_ITEM 292549067) itemoff 7706 itemsize 38
                location key (8026788 INODE_ITEM 0) type FILE
                transid 270385 data_len 0 name_len 8
                name: hdisk.pl
        item 215 key (10550 DIR_ITEM 299125689) itemoff 7660 itemsize 46
                location key (7884520 INODE_ITEM 0) type FILE
                transid 270371 data_len 0 name_len 16
                name: dbus-run-session
        item 216 key (10550 DIR_ITEM 299386369) itemoff 7622 itemsize 38
                location key (7899436 INODE_ITEM 0) type FILE
                transid 270375 data_len 0 name_len 8
                name: fipshmac
        item 217 key (10550 DIR_ITEM 300266558) itemoff 7579 itemsize 43
                location key (8375547 INODE_ITEM 0) type FILE
                transid 298231 data_len 0 name_len 13
                name: zipgrep-plain
        item 218 key (10550 DIR_ITEM 300774940) itemoff 7531 itemsize 48
                location key (8783607 INODE_ITEM 0) type FILE
                transid 336134 data_len 0 name_len 18
                name: plasma_waitforname
        item 219 key (10550 DIR_ITEM 302841040) itemoff 7498 itemsize 33
                location key (7866057 INODE_ITEM 0) type FILE
                transid 270369 data_len 0 name_len 3
                name: tgz
        item 220 key (10550 DIR_ITEM 304579196) itemoff 7461 itemsize 37
                location key (7940452 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 7
                name: lavpipe
        item 221 key (10550 DIR_ITEM 305503608) itemoff 7422 itemsize 39
                location key (7987230 INODE_ITEM 0) type FILE
                transid 270381 data_len 0 name_len 9
                name: splitmail
        item 222 key (10550 DIR_ITEM 308616482) itemoff 7383 itemsize 39
                location key (8114312 INODE_ITEM 0) type FILE
                transid 270393 data_len 0 name_len 9
                name: konqueror
        item 223 key (10550 DIR_ITEM 309391749) itemoff 7347 itemsize 36
                location key (8145072 INODE_ITEM 0) type FILE
                transid 270396 data_len 0 name_len 6
                name: ktrash
        item 224 key (10550 DIR_ITEM 310183877) itemoff 7311 itemsize 36
                location key (7935740 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 6
                name: sg_ses
        item 225 key (10550 DIR_ITEM 311147668) itemoff 7274 itemsize 37
                location key (7939276 INODE_ITEM 0) type FILE
                transid 270378 data_len 0 name_len 7
                name: pamtilt
        item 226 key (10550 DIR_ITEM 311778987) itemoff 7236 itemsize 38
                location key (7939418 INODE_ITEM 0) type SYMLINK
                transid 270378 data_len 0 name_len 8
                name: pnmscale
        item 227 key (10550 DIR_ITEM 312520839) itemoff 7195 itemsize 41
                location key (8141472 INODE_ITEM 0) type FILE
                transid 270396 data_len 0 name_len 11
                name: kcookiejar4
        item 228 key (10550 DIR_ITEM 314757239) itemoff 7156 itemsize 39
                location key (7881402 INODE_ITEM 0) type FILE
                transid 270371 data_len 0 name_len 9
                name: clrunimap
        item 229 key (10550 DIR_ITEM 317231188) itemoff 7106 itemsize 50
                location key (8038449 INODE_ITEM 0) type FILE
                transid 270386 data_len 0 name_len 20
                name: akonadi_ews_resource
localhost:/home/thekir #
Comment 22 Wenruo Qu 2018-10-18 13:59:55 UTC
(In reply to Tom Hanker from comment #21)
> (In reply to Wenruo Qu from comment #20)
> > > BTRFS critical (device sda2): corrupt leaf, slot offset bad:
> > > block=43231330304,root=1, slot=47
> > > 
> > 
> > That's what we need.
> > 
> > It's a corruption in root tree, so it's really serious.
> > 
> > We need some extra info about the corruption.
> > 
> > Would you please use the following command to dump that corrupted block?
> > 
> > # btrfs ins dump-tree -b 43231330304 /dev/sda2
> > 
> > > This message comes 3 times, then the it goes on with the message above. I
> > > red it could be a hardware crash, but if why not with kernel .16
> > 
> > Because .16 kernel doesn't has such restrict content check.
> > But strangely, the error message doesn't provide enough info, unlike latest
> > upstream.
> > (Normally it should provide more detailed info on which is going wrong).
> > 
> > Anyway, that "btrfs ins dump-tree" should tell us if it's just a minor
> > fixable problem or a real corruption.
> > 
> > Thanks,
> > Qu
> 
> Ok. the number 43231330304 was only an example, which i copy from the net. i
> bootet again and got the right one. this is the output, very long:

When pasting kernel message, please ensure it's pasted as is, without any difference.

I want to be sure that the slot number and the error message are also correct.
At least from what I see, slot 46/47/48 has nothing wrong, and the tree block owner doesn't 
even match with the output.

If the slot number is not 47, please provide the correct kernel message.
If it's slot 47, then it maybe some thing wrong with the backported tree checker.

Thanks,
Qu

> 
> btrfs ins dump-tree -b 301168066560 /dev/sda2
> btrfs-progs v4.15
> leaf 301168066560 items 230 free space 1356 generation 362173 owner 2789
> leaf 301168066560 flags 0x1(WRITTEN) backref revision 1
> fs uuid 922affe3-95d9-49e5-92aa-9fb3c4749279
> chunk uuid 858737e2-d8ab-4e61-9ca3-95dc7035c5c5

> localhost:/home/thekir #
Comment 23 Tom Hanker 2018-10-18 14:09:06 UTC
(In reply to Wenruo Qu from comment #22)
> (In reply to Tom Hanker from comment #21)
> > (In reply to Wenruo Qu from comment #20)
> > > > BTRFS critical (device sda2): corrupt leaf, slot offset bad:
> > > > block=43231330304,root=1, slot=47
> > > > 
> > > 
> > > That's what we need.
> > > 
> > > It's a corruption in root tree, so it's really serious.
> > > 
> > > We need some extra info about the corruption.
> > > 
> > > Would you please use the following command to dump that corrupted block?
> > > 
> > > # btrfs ins dump-tree -b 43231330304 /dev/sda2
> > > 
> > > > This message comes 3 times, then the it goes on with the message above. I
> > > > red it could be a hardware crash, but if why not with kernel .16
> > > 
> > > Because .16 kernel doesn't has such restrict content check.
> > > But strangely, the error message doesn't provide enough info, unlike latest
> > > upstream.
> > > (Normally it should provide more detailed info on which is going wrong).
> > > 
> > > Anyway, that "btrfs ins dump-tree" should tell us if it's just a minor
> > > fixable problem or a real corruption.
> > > 
> > > Thanks,
> > > Qu
> > 
> > Ok. the number 43231330304 was only an example, which i copy from the net. i
> > bootet again and got the right one. this is the output, very long:
> 
> When pasting kernel message, please ensure it's pasted as is, without any
> difference.
> 
> I want to be sure that the slot number and the error message are also
> correct.
> At least from what I see, slot 46/47/48 has nothing wrong, and the tree
> block owner doesn't 
> even match with the output.
> 
> If the slot number is not 47, please provide the correct kernel message.
> If it's slot 47, then it maybe some thing wrong with the backported tree
> checker.
> 
> Thanks,
> Qu
> 
> > 
> > btrfs ins dump-tree -b 301168066560 /dev/sda2
> > btrfs-progs v4.15
> > leaf 301168066560 items 230 free space 1356 generation 362173 owner 2789
> > leaf 301168066560 flags 0x1(WRITTEN) backref revision 1
> > fs uuid 922affe3-95d9-49e5-92aa-9fb3c4749279
> > chunk uuid 858737e2-d8ab-4e61-9ca3-95dc7035c5c5
> 
> > localhost:/home/thekir #

The slotnumber slot=75 ino 10550
Comment 24 Wenruo Qu 2018-10-18 14:12:11 UTC
> > > localhost:/home/thekir #
> 
> The slotnumber slot=75 ino 10550

This makes sense. It's DIR_ITEM with conflicting hash.
I'll check if the kernel has something wrong in the tree checker.

Thanks,
Qu
Comment 25 Tom Hanker 2018-10-18 16:40:57 UTC
(In reply to Wenruo Qu from comment #24)
> > > > localhost:/home/thekir #
> > 
> > The slotnumber slot=75 ino 10550
> 
> This makes sense. It's DIR_ITEM with conflicting hash.
> I'll check if the kernel has something wrong in the tree checker.
> 
> Thanks,
> Qu

O.k thats fine. Thank you!
Comment 26 Wenruo Qu 2018-10-19 00:45:23 UTC
OK, it turns out that, it's a real corruption here.

In slot 75:

        item 75 key (10550 DIR_ITEM 108382495) itemoff 13225 itemsize 72
                location key (74333 INODE_ITEM 0) type FILE
                transid 58 data_len 0 name_len 6
                name: newg
                location key (8869339 INODE_ITEM 0) type FILE
                transid 346840 data_len 0 name_len 6
                name: newgrp

The first DIR_ITEM, name_len 6 with name: "newg\0\0".
Its hash won't match with 108382495.

And .16 kernel detects such problem, then refuse to read it.
(early kernel doesn't has such error detection, so they just let them pass)

Considering the transid, it's pretty old, so could be caused by early kernel bugs.

In that case, we need to ensure there is only one corruption for the fs.

Please run "btrfs check --readonly" on that fs, and attach the result as a file.

After that, we can determine how to solve the problem.

Thanks,
Qu
Comment 27 Tom Hanker 2018-10-19 06:02:08 UTC
Created attachment 786498 [details]
btrfs --check
Comment 28 Tom Hanker 2018-10-19 06:04:04 UTC
I used --forece because i haven´t much time, so the check was within a running system.
Comment 29 Wenruo Qu 2018-10-19 07:19:04 UTC
Just as the btrfs check shows, your filesystem is indeed corrupted.

And the corruption is caused by old kernel, and the corruption is shared between several snapshots.

Currently btrfs check can detect it but can't really fix it well.
I'll enhance btrfs check repair functionality to fix it.

Thanks,
Qu
Comment 30 Tom Hanker 2018-10-19 10:18:44 UTC
(In reply to Wenruo Qu from comment #29)
> Just as the btrfs check shows, your filesystem is indeed corrupted.
> 
> And the corruption is caused by old kernel, and the corruption is shared
> between several snapshots.
> 
> Currently btrfs check can detect it but can't really fix it well.
> I'll enhance btrfs check repair functionality to fix it.
> 
> Thanks,
> Qu

Ok. thank you! Could the error caused by an update, dist-upgrade, cloning? I upgraded from 42.3 to 15.0 and later clone it to a new ssd.
Comment 31 Wenruo Qu 2018-10-19 11:41:28 UTC
(In reply to Tom Hanker from comment #30)
> (In reply to Wenruo Qu from comment #29)
> > Just as the btrfs check shows, your filesystem is indeed corrupted.
> > 
> > And the corruption is caused by old kernel, and the corruption is shared
> > between several snapshots.
> > 
> > Currently btrfs check can detect it but can't really fix it well.
> > I'll enhance btrfs check repair functionality to fix it.
> > 
> > Thanks,
> > Qu
> 
> Ok. thank you! Could the error caused by an update, dist-upgrade, cloning? I
> upgraded from 42.3 to 15.0 and later clone it to a new ssd.

So it looks like old kernel caused the problem.

I'm not sure if dist-upgrade could cause such problem, but the bug itself looks like something went wrong with rename/overwrite.

At least it shouldn't happen for newer kernel.

You could still use older kernel to access the files.
But it's recommended to try to copy your subvolume contents to a newly created subvolume using reflink. Then delete the following subvolumes:
598
2702
2703
2711
2713
2719
2730
2740
2741
2783
2784
2788
2789
2835
2836
2837
2838
2839
2840
2841
2842
2843
2844
2845
2846
2847
2848
2849
2850

With newly created subvolume and copied files, that corrupted item should not go into the new subvolume.
And with old subvolumes/snapshots removed, there will be no corruption remain.

Thanks,
Qu
Comment 32 Tom Hanker 2018-10-19 13:53:09 UTC
(In reply to Wenruo Qu from comment #31)
> (In reply to Tom Hanker from comment #30)
> > (In reply to Wenruo Qu from comment #29)
> > > Just as the btrfs check shows, your filesystem is indeed corrupted.
> > > 
> > > And the corruption is caused by old kernel, and the corruption is shared
> > > between several snapshots.
> > > 
> > > Currently btrfs check can detect it but can't really fix it well.
> > > I'll enhance btrfs check repair functionality to fix it.
> > > 
> > > Thanks,
> > > Qu
> > 
> > Ok. thank you! Could the error caused by an update, dist-upgrade, cloning? I
> > upgraded from 42.3 to 15.0 and later clone it to a new ssd.
> 
> So it looks like old kernel caused the problem.
> 
> I'm not sure if dist-upgrade could cause such problem, but the bug itself
> looks like something went wrong with rename/overwrite.
> 
> At least it shouldn't happen for newer kernel.
> 
> You could still use older kernel to access the files.
> But it's recommended to try to copy your subvolume contents to a newly
> created subvolume using reflink. Then delete the following subvolumes:
> 598
> 2702
> 2703
> 2711
> 2713
> 2719
> 2730
> 2740
> 2741
> 2783
> 2784
> 2788
> 2789
> 2835
> 2836
> 2837
> 2838
> 2839
> 2840
> 2841
> 2842
> 2843
> 2844
> 2845
> 2846
> 2847
> 2848
> 2849
> 2850
> 
> With newly created subvolume and copied files, that corrupted item should
> not go into the new subvolume.
> And with old subvolumes/snapshots removed, there will be no corruption
> remain.
> 
> Thanks,
> Qu

Thank you! But which subvolume should i copy? First delete than copy? I got to many? What is a reflink, i only see a command: btrfs subvolume delete do the command need a part or the number, or can i delete it with snapper? 

this is my list: 

btrfs subvolume list /
ID 257 gen 356300 top level 5 path @
ID 258 gen 363922 top level 257 path @/.snapshots
ID 260 gen 363591 top level 257 path @/boot/grub2/i386-pc
ID 261 gen 360656 top level 257 path @/boot/grub2/x86_64-efi
ID 262 gen 362998 top level 257 path @/opt
ID 263 gen 362173 top level 257 path @/srv
ID 264 gen 363956 top level 257 path @/tmp
ID 265 gen 363823 top level 257 path @/usr/local
ID 266 gen 363823 top level 257 path @/var/cache
ID 267 gen 350923 top level 257 path @/var/crash
ID 268 gen 363702 top level 257 path @/var/lib/libvirt/images
ID 269 gen 68 top level 257 path @/var/lib/machines
ID 270 gen 277200 top level 257 path @/var/lib/mailman
ID 271 gen 277200 top level 257 path @/var/lib/mariadb
ID 272 gen 277200 top level 257 path @/var/lib/mysql
ID 273 gen 277200 top level 257 path @/var/lib/named
ID 274 gen 277200 top level 257 path @/var/lib/pgsql
ID 275 gen 363956 top level 257 path @/var/log                                                                                         
ID 276 gen 350923 top level 257 path @/var/opt                                                                                         
ID 277 gen 363946 top level 257 path @/var/spool                                                                                       
ID 278 gen 363624 top level 257 path @/var/tmp                                                                                         
ID 540 gen 277200 top level 258 path @/.snapshots/221/snapshot                                                                         
ID 596 gen 350923 top level 258 path @/.snapshots/268/snapshot                                                                         
ID 598 gen 355469 top level 258 path @/.snapshots/270/snapshot                                                                         
ID 2702 gen 356406 top level 258 path @/.snapshots/564/snapshot                                                                        
ID 2703 gen 360656 top level 258 path @/.snapshots/565/snapshot                                                                        
ID 2711 gen 360656 top level 258 path @/.snapshots/566/snapshot                                                                        
ID 2713 gen 360656 top level 258 path @/.snapshots/568/snapshot                                                                        
ID 2719 gen 360656 top level 258 path @/.snapshots/570/snapshot                                                                        
ID 2730 gen 360656 top level 258 path @/.snapshots/579/snapshot                                                                        
ID 2740 gen 360656 top level 258 path @/.snapshots/589/snapshot
ID 2741 gen 360663 top level 258 path @/.snapshots/590/snapshot
ID 2783 gen 361133 top level 258 path @/.snapshots/629/snapshot
ID 2784 gen 361143 top level 258 path @/.snapshots/630/snapshot
ID 2788 gen 361232 top level 258 path @/.snapshots/633/snapshot
ID 2789 gen 363956 top level 258 path @/.snapshots/634/snapshot
ID 2841 gen 363083 top level 258 path @/.snapshots/685/snapshot
ID 2842 gen 363085 top level 258 path @/.snapshots/686/snapshot
ID 2843 gen 363140 top level 258 path @/.snapshots/687/snapshot
ID 2844 gen 363142 top level 258 path @/.snapshots/688/snapshot
ID 2845 gen 363552 top level 258 path @/.snapshots/689/snapshot
ID 2846 gen 363553 top level 258 path @/.snapshots/690/snapshot
ID 2847 gen 363580 top level 258 path @/.snapshots/691/snapshot
ID 2848 gen 363587 top level 258 path @/.snapshots/692/snapshot
ID 2849 gen 363588 top level 258 path @/.snapshots/693/snapshot
ID 2850 gen 363590 top level 258 path @/.snapshots/694/snapshot
Comment 33 Wenruo Qu 2018-10-19 14:05:14 UTC
> 
> Thank you! But which subvolume should i copy?

I'm not familiar with openSUSE's snapshots/subvolumes layout.

Maybe Jeff could provide some suggestion?

> First delete than copy?

Copy first, then delete.

> I got to many?

The corruption happens between snapshot 596 and 598.
If you coud provide the timestamp of that two snapshots, we could get a basic idea of when the corruption happens (we're just curious when the corruption happens).

> What is a reflink,

# cp --reflink=always

It hugely speedup file copying as it skips real file IO, and only inserts data pointers.

And since you're using btrfs for your root, it's recommended to do the copy using rescue ISO.

> i only see a command: btrfs subvolume delete do
> the command need a part or the number, or can i delete it with snapper? 

Unfortunately I'm not a expert of snapper and its snapshots/subvolume layout.
So you may need help from Jeff.

Thanks,
Qu
Comment 34 Tom Hanker 2018-10-19 18:00:07 UTC
(In reply to Wenruo Qu from comment #33)
> > 
> > Thank you! But which subvolume should i copy?
> 
> I'm not familiar with openSUSE's snapshots/subvolumes layout.
> 
> Maybe Jeff could provide some suggestion?
> 
> > First delete than copy?
> 
> Copy first, then delete.
> 
> > I got to many?
> 
> The corruption happens between snapshot 596 and 598.
> If you coud provide the timestamp of that two snapshots, we could get a
> basic idea of when the corruption happens (we're just curious when the
> corruption happens).
> 
> > What is a reflink,
> 
> # cp --reflink=always
> 
> It hugely speedup file copying as it skips real file IO, and only inserts
> data pointers.
> 
> And since you're using btrfs for your root, it's recommended to do the copy
> using rescue ISO.
> 
> > i only see a command: btrfs subvolume delete do
> > the command need a part or the number, or can i delete it with snapper? 
> 
> Unfortunately I'm not a expert of snapper and its snapshots/subvolume layout.
> So you may need help from Jeff.
> 
> Thanks,
> Qu

is there a possibility to show the timestamp? snapper shows the date and the time, but the two numbers aren´t available anymore in snapper. unfortunaltyl i haven´t any idea about filesystems.
Comment 35 Wenruo Qu 2018-10-20 01:38:43 UTC
> is there a possibility to show the timestamp? snapper shows the date and the
> time, but the two numbers aren´t available anymore in snapper. unfortunaltyl
> i haven´t any idea about filesystems.

You could just use "ls -alhd" to show the mtime of that directory.

Thanks,
Qu
Comment 36 Tom Hanker 2018-10-20 06:35:01 UTC
(In reply to Wenruo Qu from comment #35)
> > is there a possibility to show the timestamp? snapper shows the date and the
> > time, but the two numbers aren´t available anymore in snapper. unfortunaltyl
> > i haven´t any idea about filesystems.
> 
> You could just use "ls -alhd" to show the mtime of that directory.
> 
> Thanks,
> Qu

ls -alhd 270
drwxr-xr-x 1 root root 16  2. Jun 2017  270

ls -alhd 264
drwxr-xr-x 1 root root 0  2. Jun 2017  264

I assume, that i made an dist-upgrade, maybe with this upgrade there was a change from ext4 to btrfs. I found out by chance, that i have btrfs a few month ago, when my ssd was full because of the snapshots. 



How can i delete it? 
Wih the id it doesn´t work:

btrfs subvolume delete 2850
ERROR: cannot access subvolume 2850: No such file or directory

The same here:

btrfs subvolume delete @/.snapshots/694/snapshot
ERROR: cannot access subvolume @/.snapshots/694/snapshot: No such file or directory

I can delete it with snapper snapper delete 694 
But this:

D 598 gen 355469 top level 258 path @/.snapshots/270/snapshot

snapper delete 270, can´t find the file.
Comment 37 Wenruo Qu 2018-10-22 04:51:34 UTC
> 
> How can i delete it? 
> Wih the id it doesn´t work:
> 
> btrfs subvolume delete 2850
> ERROR: cannot access subvolume 2850: No such file or directory

Please refer "btrfs-subvolume"(8) man page.

"btrfs subvolume delete" doesn't support subvolumme id.

> 
> The same here:
> 
> btrfs subvolume delete @/.snapshots/694/snapshot
> ERROR: cannot access subvolume @/.snapshots/694/snapshot: No such file or
> directory

That '@' is not a indicator.

You should use "btrfs subvolume delete /.snapshots/694/snapshot"

Thanks,
Qu

> 
> I can delete it with snapper snapper delete 694 
> But this:
> 
> D 598 gen 355469 top level 258 path @/.snapshots/270/snapshot
> 
> snapper delete 270, can´t find the file.
Comment 38 Tom Hanker 2018-10-22 09:55:33 UTC
During the weekend i found out, that with snapper you can undo changes between two snapshots. But i tried do do it between snapshot 268..270, but snapper can´t found them. 

snapper -v undochange xxx..xxx

https://www.suse.com/documentation/sled11/book_sle_admin/data/sec_snapper_auto.html
Comment 39 Tom Hanker 2018-10-23 11:22:30 UTC
I managed to delete the subvolumes, but the error still exists. I can´t boot with kernel .19. Can i try to do btrfs check --repair or is it better to make a new instalation?
Comment 40 Wenruo Qu 2018-10-23 11:45:21 UTC
(In reply to Tom Hanker from comment #39)
> I managed to delete the subvolumes, but the error still exists. I can´t boot
> with kernel .19. Can i try to do btrfs check --repair or is it better to
> make a new instalation?

You have to do the btrfs check --readonly -> delete corrupted subvolumes -> btrfs check --readonly
loop until all corrupted filesystems are deleted.

However the problem is, your current root fs also contains the corrupted leaf, through all the snapshots.

That's why I'm asking you to copy the contents of your root fs to a new subvolume, to avoid inheriting the corrupted leaf.

If above workaround is too complex, you could do a new installtion.

Thanks,
Qu
Comment 41 Jeff Mahoney 2018-10-23 17:05:24 UTC
Created attachment 786911 [details]
btrfs check output, in text format.

Please upload text in text files instead of open office docs.
Comment 42 Jeff Mahoney 2018-10-23 17:26:22 UTC
As best I can tell, it looks like btrfs check --repair can't deal with this file system yet.  We'd view that as a bug since you've encountered corruption it can't handle.  Before you remove the broken snapshots, could you do:

btrfs-image <srcdev> <some file> while the file system is unmounted and post the results somewhere so we can download it?  It won't share any of your data but it will share your filenames.  Normally we'd suggest using the "sanitize" option to scrub the filenames but since this is specific to directory entries, it wouldn't work to resolve the issue.

<some file> can be a file on a different file system but it could also be "-" such that you could send it to another machine via SSH or something, e.g.

btrfs-image <srcdev> - | ssh somehost "cat > btrfs.img"

Then, once you've captured that, you can run:

# snapper delete 270-694

The btrfs check output shows that the corrupted directory entry exists in those snapshots but not the root volume, so you should be ok without having to rebuild your file system from nothing.  It will remove those snapshots.
Comment 43 Tom Hanker 2018-10-25 09:59:09 UTC
(In reply to Jeff Mahoney from comment #42)
> As best I can tell, it looks like btrfs check --repair can't deal with this
> file system yet.  We'd view that as a bug since you've encountered
> corruption it can't handle.  Before you remove the broken snapshots, could
> you do:
> 
> btrfs-image <srcdev> <some file> while the file system is unmounted and post
> the results somewhere so we can download it?  It won't share any of your
> data but it will share your filenames.  Normally we'd suggest using the
> "sanitize" option to scrub the filenames but since this is specific to
> directory entries, it wouldn't work to resolve the issue.
> 
> <some file> can be a file on a different file system but it could also be
> "-" such that you could send it to another machine via SSH or something, e.g.
> 
> btrfs-image <srcdev> - | ssh somehost "cat > btrfs.img"
> 
> Then, once you've captured that, you can run:
> 
> # snapper delete 270-694
> 
> The btrfs check output shows that the corrupted directory entry exists in
> those snapshots but not the root volume, so you should be ok without having
> to rebuild your file system from nothing.  It will remove those snapshots.

Thank you for your help. I tried it yesterday and someway the whole system crasched. Then i decide to make a clean install. Now all is o.k. The new Kernel is working.
Comment 44 Wenruo Qu 2019-02-25 04:58:46 UTC
Since no further update, close this bug.