Bug 1111373 - INTERNAL ERROR: Signal 11 in pid 2026 (4.9.1-git.46.ff87d06baceSUSE-oS15.5-x86_64)
INTERNAL ERROR: Signal 11 in pid 2026 (4.9.1-git.46.ff87d06baceSUSE-oS15.5-x8...
Status: RESOLVED FIXED
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Samba
Current
Other Other
: P5 - None : Normal (vote)
: ---
Assigned To: David Mulder
The 'Opening Windows to a Wider World' guys
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2018-10-10 11:42 UTC by Philip Raets
Modified: 2018-12-20 15:53 UTC (History)
3 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---
david.mulder: needinfo? (philip.raets)


Attachments
smb service log (2.75 KB, text/x-log)
2018-10-10 11:42 UTC, Philip Raets
Details
winbind log (2.84 KB, text/x-log)
2018-10-10 11:43 UTC, Philip Raets
Details
smb config (1.06 KB, text/plain)
2018-10-10 11:45 UTC, Philip Raets
Details
winbind log (2.84 KB, text/x-log)
2018-10-11 20:22 UTC, Philip Raets
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Philip Raets 2018-10-10 11:42:54 UTC
Created attachment 785614 [details]
smb service log

I log on with a domain user in Tumbleweed, but after the upgrade to samba 4.9.1  the smb and winbind services don't start at boot, they crash. I have to log on with a local user, start the services manually, and then they start fine. After this I can log on with a domain user.

Nothing changed in smb.conf
Comment 1 Philip Raets 2018-10-10 11:43:21 UTC
Created attachment 785615 [details]
winbind log
Comment 2 Philip Raets 2018-10-10 11:45:35 UTC
Created attachment 785616 [details]
smb config
Comment 3 David Mulder 2018-10-11 19:09:20 UTC
Can you set:
log level = 2 winbind:10
in your smb.conf and get the winbind debug from startup again?
Comment 4 Philip Raets 2018-10-11 20:22:26 UTC
Created attachment 785785 [details]
winbind log

This is the winbind log after setting the loglevel = 2 winbind:10 in smb.conf
Comment 5 David Mulder 2018-10-12 13:21:50 UTC
(In reply to Philip Raets from comment #4)
> Created attachment 785785 [details]
> winbind log
> 
> This is the winbind log after setting the loglevel = 2 winbind:10 in smb.conf

Well, that didn't provide any additional debug. I'll have to see if I can reproduce it.
Comment 6 Philip Raets 2018-10-23 12:25:10 UTC
I just found out this happens when I get an IP from the DHCP-server and samba is started before an ip-address is given.

If I have a fixed IP, samba doesn't crash (but don't want to have a fixed ip).

My network:

bridge0: connected to Boma
        "bridge0"
        bridge, D4:BE:D9:A1:09:C8, sw, mtu 1500
        ip4 default
        inet4 192.168.10.85/24
        route4 0.0.0.0/0
        route4 192.168.10.0/24

eth0: connected to bridge0 slave 1
        "Intel 82579LM"
        ethernet (e1000e), D4:BE:D9:A1:09:C8, hw, mtu 1500
        master bridge0
        route6 ff00::/8

vnet0: connected to vnet0
        "vnet0"
        tun, FE:54:00:99:08:3C, sw, mtu 1500
        master bridge0

lo: unmanaged
        "lo"
        loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536
Comment 7 David Mulder 2018-10-23 13:46:24 UTC
Winbind is configured to not start until the network-online target is reached, though what this means depends on the network service.
Which network service are you using (wicked/networkmanager)?
Comment 8 Philip Raets 2018-10-23 13:50:47 UTC
I am using NetworkManager (bridge created with nm-connection-editor, which is used for kvm)
Comment 9 David Mulder 2018-10-23 13:54:42 UTC
Also, try adding this line to the Service section of /usr/lib/systemd/system/winbind.service:
ExecStartPre=/bin/sh -c 'until ping -c1 google.com; do sleep 1; done;'

This is just to test if winbind starts if we delay until you get a dhcp address. I suspect you have a slow dhcp server (although winbind should not crash when you haven't gotten an address yet).
Comment 10 Philip Raets 2018-10-23 14:37:19 UTC
After adding the line, winbind now waits and starts without crashing (was pinging an internal address):

PRAETS:/home/BOMA/praets # journalctl -b -u winbind.service 
-- Logs begin at Wed 2016-12-21 15:46:32 CET, end at Tue 2018-10-23 16:33:11 CEST. --
okt 23 16:26:52 PRAETS.boma.local systemd[1]: Starting Samba Winbind Daemon...
okt 23 16:26:52 PRAETS.boma.local sh[4118]: connect: Network is unreachable
okt 23 16:26:53 PRAETS.boma.local sh[4118]: connect: Network is unreachable
okt 23 16:26:54 PRAETS.boma.local sh[4118]: connect: Network is unreachable
okt 23 16:26:55 PRAETS.boma.local sh[4118]: connect: Network is unreachable
okt 23 16:26:56 PRAETS.boma.local sh[4118]: connect: Network is unreachable
okt 23 16:26:57 PRAETS.boma.local sh[4118]: connect: Network is unreachable
okt 23 16:26:58 PRAETS.boma.local sh[4118]: connect: Network is unreachable
okt 23 16:26:59 PRAETS.boma.local sh[4118]: connect: Network is unreachable
okt 23 16:27:00 PRAETS.boma.local sh[4118]: connect: Network is unreachable
okt 23 16:27:01 PRAETS.boma.local sh[4118]: PING 10.10.255.2 (10.10.255.2) 56(84) bytes of data.
okt 23 16:27:01 PRAETS.boma.local sh[4118]: 64 bytes from 10.10.255.2: icmp_seq=1 ttl=64 time=0.608 ms
okt 23 16:27:01 PRAETS.boma.local sh[4118]: --- 10.10.255.2 ping statistics ---
okt 23 16:27:01 PRAETS.boma.local sh[4118]: 1 packets transmitted, 1 received, 0% packet loss, time 0ms
okt 23 16:27:01 PRAETS.boma.local sh[4118]: rtt min/avg/max/mdev = 0.608/0.608/0.608/0.000 ms
okt 23 16:27:02 PRAETS.boma.local winbindd[4334]: [2018/10/23 16:27:02.014722,  0, pid=4334, effective(0, 0), real(0, 0)] ../lib/util/become_daemon.c:138(daemon_ready)
okt 23 16:27:02 PRAETS.boma.local winbindd[4334]:   daemon_ready: STATUS=daemon 'winbindd' finished starting up and ready to serve connections
okt 23 16:27:02 PRAETS.boma.local systemd[1]: Started Samba Winbind Daemon.
Comment 11 David Mulder 2018-10-23 19:36:41 UTC
Can you enable the repo-debug repo, then install samba-winbind-debuginfo? Also install gdb for getting a backtrace:
sudo zypper mr -e repo-debug && sudo zypper ref
sudo zypper in samba-winbind-debuginfo gdb

Reproduce the failure, then get a backtrace from systemd:

# coredumpctl gdb winbindd
(gdb) bt
#0  0x00007fbc00b8508b in raise () from /lib64/libc.so.6
etc...
Comment 12 Philip Raets 2018-10-24 09:44:26 UTC
this is the output:

#0  0x00007f7a5909a08b in raise () from /lib64/libc.so.6
#1  0x00007f7a590834e9 in abort () from /lib64/libc.so.6
#2  0x00007f7a59b63bf3 in dump_core () from /usr/lib64/libsmbconf.so.0
#3  0x00007f7a59b54beb in smb_panic_s3 () from /usr/lib64/libsmbconf.so.0
#4  0x00007f7a5a86231f in smb_panic () from /usr/lib64/libsamba-util.so.0
#5  0x00007f7a5a862556 in ?? () from /usr/lib64/libsamba-util.so.0
#6  <signal handler called>
#7  0x00007f7a59105374 in __memmove_sse2_unaligned_erms () from /lib64/libc.so.6
#8  0x00007f7a5a860ed1 in smb_memdup () from /usr/lib64/libsamba-util.so.0
#9  0x00007f7a59b652f0 in load_interfaces () from /usr/lib64/libsmbconf.so.0
#10 0x00005634d7cc48f0 in reload_services_file (lfile=0x0) at ../source3/winbindd/winbindd.c:124
#11 0x00005634d7cc1221 in main (argc=<optimized out>, argv=<optimized out>) at ../source3/winbindd/winbindd.c:1752
Comment 13 David Mulder 2018-10-24 14:25:00 UTC
(In reply to Philip Raets from comment #12)
> this is the output:
> 
> #0  0x00007f7a5909a08b in raise () from /lib64/libc.so.6
> #1  0x00007f7a590834e9 in abort () from /lib64/libc.so.6
> #2  0x00007f7a59b63bf3 in dump_core () from /usr/lib64/libsmbconf.so.0
> #3  0x00007f7a59b54beb in smb_panic_s3 () from /usr/lib64/libsmbconf.so.0
> #4  0x00007f7a5a86231f in smb_panic () from /usr/lib64/libsamba-util.so.0
> #5  0x00007f7a5a862556 in ?? () from /usr/lib64/libsamba-util.so.0
> #6  <signal handler called>
> #7  0x00007f7a59105374 in __memmove_sse2_unaligned_erms () from
> /lib64/libc.so.6
> #8  0x00007f7a5a860ed1 in smb_memdup () from /usr/lib64/libsamba-util.so.0
> #9  0x00007f7a59b652f0 in load_interfaces () from /usr/lib64/libsmbconf.so.0
> #10 0x00005634d7cc48f0 in reload_services_file (lfile=0x0) at
> ../source3/winbindd/winbindd.c:124
> #11 0x00005634d7cc1221 in main (argc=<optimized out>, argv=<optimized out>)
> at ../source3/winbindd/winbindd.c:1752

Perfect. That's what I was looking for.
Comment 14 David Mulder 2018-10-24 15:55:59 UTC
(In reply to Philip Raets from comment #0)
> Created attachment 785614 [details]
> smb service log
> 
> I log on with a domain user in Tumbleweed, but after the upgrade to samba
> 4.9.1  the smb and winbind services don't start at boot, they crash. I have
> to log on with a local user, start the services manually, and then they
> start fine. After this I can log on with a domain user.
> 
> Nothing changed in smb.conf

What version of samba did you upgrade from? 4.8.x? 4.7.x?
Comment 15 David Mulder 2018-10-24 17:07:46 UTC
I found a recent change that causes load_interfaces() to memdup NULL when only 1 interface is present (such as when the only interface that's up is lo), this would cause a segfault. I suspect that may be what we're seeing here.

I'm building a test package and would like you to test it when it's ready (details will follow).
Comment 16 David Mulder 2018-10-24 18:23:10 UTC
The test build is running here:
https://build.opensuse.org/package/show/home:dmulder:4.9.1-get_interfaces-segfault/samba

When the build completes, you should be able to install and test it by adding my repository and duping it:
sudo zypper ar https://download.opensuse.org/repositories/home:/dmulder:/4.9.1-get_interfaces-segfault/openSUSE_Tumbleweed dmulder-samba
sudo zypper dup -r dmulder-samba
Comment 17 Philip Raets 2018-10-25 08:13:25 UTC
I just zypper dup once or twice a week, so it upgraded from the previous version of samba in Tumbleweed.

I updated to your packages, and they work, winbind and samba are started at boot.

Maybe unrelated:
but I have a virtual machine (with KVM) which is set to autostart at boot, and this fails now to start at boot. (I can start it after I log in)

If I revert the packages back to the original ones, and apply the workaround in /usr/lib/systemd/system/winbind.service, my virtual machine starts fine at boot.
Comment 18 David Mulder 2018-10-25 19:01:08 UTC
(In reply to Philip Raets from comment #17)
> Maybe unrelated:
> but I have a virtual machine (with KVM) which is set to autostart at boot,
> and this fails now to start at boot. (I can start it after I log in)
> 
> If I revert the packages back to the original ones, and apply the workaround
> in /usr/lib/systemd/system/winbind.service, my virtual machine starts fine
> at boot.

Is there crash at startup, or is it simply not starting the process? You say you revert the packages, what do you mean? Are you reverting 4.9.1 to 4.8.x? Are you talking about the test packages I built for you? You didn't specify.

Could you collect debug, etc? If there is a crash, follow the steps I gave you before and collect a crash dump.
Comment 19 David Mulder 2018-10-25 19:04:41 UTC
(In reply to David Mulder from comment #18)
> (In reply to Philip Raets from comment #17)
> > Maybe unrelated:
> > but I have a virtual machine (with KVM) which is set to autostart at boot,
> > and this fails now to start at boot. (I can start it after I log in)
> > 
> > If I revert the packages back to the original ones, and apply the workaround
> > in /usr/lib/systemd/system/winbind.service, my virtual machine starts fine
> > at boot.
> 
> Is there crash at startup, or is it simply not starting the process? You say
> you revert the packages, what do you mean? Are you reverting 4.9.1 to 4.8.x?
> Are you talking about the test packages I built for you? You didn't specify.
> 
> Could you collect debug, etc? If there is a crash, follow the steps I gave
> you before and collect a crash dump.

Oh, I just realized you mean the *VM* doesn't start at boot, not *samba*. It's probably a related issue. I'd guess there's a bug in network manager that is causing processes to start before network up. It just happened to bring a different bug in samba to the surface. You should file a separate bug against network manager.
Comment 20 David Mulder 2018-10-25 19:11:18 UTC
I've posted a bug upstream here:
https://bugzilla.samba.org/show_bug.cgi?id=13665

And the patch has been reviewed and should land in master soon. It should end up in 4.9.1 (tumbleweed) shortly after.
Comment 22 David Mulder 2018-12-20 15:53:11 UTC
Fixed as of 4.9.2