Bugzilla – Bug 1143125
On i386, systemd-vconsole-setup fails due to MemoryDenyWriteExecute=yes used in systemd-udevd.service
Last modified: 2021-07-02 07:47:51 UTC
Created attachment 811849 [details] journalctl Booting a freshly installed Tumbleweed system with the following vconsole.conf results in sequential success and failure (seen in attached journalctl) # cat /etc/vconsole.conf KEYMAP=us FONT=eurlatgr.psfu FONT_MAP= FONT_UNIMAP=
Please attach the output of "journalctl -b -o short-monotonic" with the issue reproduced.
Created attachment 812001 [details] journalctl -b -o short-monotonic | grep -i vconsole -B 5 -A 5 That seems to contain too much info which is both unrelated to this bug and also not secure to share (e.g. you don't need to know the port on which I am running SSH and things like that). So instead I am attaching: journalctl -b -o short-monotonic | grep -i vconsole -B 5 -A 5 In case you are still missing anything specific please let me know.
Apparently systemd-vconsole-setup is only failing when spawned by udev: > [ 40.008060] pc systemd-vconsole-setup[1010]: /usr/bin/setfont failed with exit status 65. > [ 40.026289] pc systemd-vconsole-setup[1011]: /usr/bin/setfont failed with exit status 65. > [ 40.100634] pc systemd-vconsole-setup[1010]: Setting source virtual console failed, ignoring remaining ones > [ 40.101557] pc systemd-udevd[858]: Process '/usr/lib/systemd/systemd-vconsole-setup' failed with exit code 1. > [ 40.128793] pc systemd-vconsole-setup[1011]: Setting source virtual console failed, ignoring remaining ones > [ 40.133664] pc systemd-udevd[880]: Process '/usr/lib/systemd/systemd-vconsole-setup' failed with exit code 1. Exit status 65 means (defined in /usr/include/sysexits.h): * EX_DATAERR -- The input data was incorrect in some way. * This should only be used for user's data & not * system files. Not sure what it means exactly for kbd but it might indicate something wrong with the font data, which is strange since I have the same vconsole.conf (so I'm using the same font) and can't see the issue. Which version of systemd are you running ? Can you try: rpm -V kbd ? Can you also try to trigger the udev rule manually with udev debug logs enabled ? # udevadm control --log-priority debug # udevadm trigger --action=add -s vtconsole" ? does it also fail ? and show the debug logs with: # journalctl -b -o short-monotonic -u systemd-udevd PRIORITY=7 Thanks.
Created attachment 812156 [details] journalctl -b -o short-monotonic -u systemd-udevd PRIORITY=7 # rpm -q systemd systemd-242-3.1.i586 # rpm -V kbd # (i.e. nothing shows) The requested journal command output is attached too. Please note that this is a freshly installed system (Tumblweed with XFCE) with English US setting during the installation and nothing has been done on it except keeping it up to date. On the Leap systems which I manage (and which are also installed with with English US setting) I see that: # cat /etc/vconsole.conf KEYMAP=us FONT=lat9w-16.psfu FONT_MAP=trivial Not sure if this is important but I thought I should mention it.
(In reply to Suse User from comment #4) > Created attachment 812156 [details] > journalctl -b -o short-monotonic -u systemd-udevd PRIORITY=7 > Thanks however please drop "PRIORITY=7" so I can see the other messages from udevd just to make sure that is still the same issue.
Created attachment 812174 [details] journalctl -b -o short-monotonic -u systemd-udevd There it is.
So the debug logs showed: > [ 133.029058] pc systemd-udevd[2255]: '/usr/lib/systemd/systemd-vconsole-setup'(err) 'gzip: error while loading shared libraries: cannot make segment writable for relocation: Operation not permitted' > [ 133.032608] pc systemd-udevd[2255]: '/usr/lib/systemd/systemd-vconsole-setup'(err) 'Bad character height 0' > [ 133.032926] pc systemd-vconsole-setup[2257]: /usr/bin/setfont failed with exit status 65. It sounds like there was an issue while loading some shared libraries when systemd-vconsole-setup executed setfont.... In /usr/lib/systemd/system/systemd-udevd.service, can you try to comment the following line and see if it helps (simplest is to reboot after the change) ? MemoryDenyWriteExecute=yes
BTW, did you activate or change any security features (such as selinux, apparmor) ?
Dominique, are you aware of any changes related to package building mode, linker flags, ... ?
(In reply to Franck Bui from comment #7) > > It sounds like there was an issue while loading some shared libraries when > systemd-vconsole-setup executed setfont.... > Actually the prob happens when setfont executes "gzip -d ..."
> BTW, did you activate or change any security features (such as selinux, apparmor) ? No. The only change I have made security-wise after the installation was to set in Yast->Security and Users->Miscellaneous Settings->File permissions from "Easy" to "Secure". But even before doing this the error messages related to vconsole were in the journal. Commenting out the line "MemoryDenyWriteExecute=yes" results in sucessfull messages only: # journalctl -b | grep vconsole Aug 01 18:54:45 acer systemd[1]: systemd-vconsole-setup.service: Succeeded. Aug 01 18:54:50 acer systemd[1]: systemd-vconsole-setup.service: Succeeded. Aug 01 18:55:00 acer systemd[1]: systemd-vconsole-setup.service: Succeeded. Aug 01 18:55:12 acer systemd[1]: systemd-vconsole-setup.service: Succeeded. Aug 01 18:55:22 acer systemd[1]: systemd-vconsole-setup.service: Succeeded. Aug 01 18:55:51 acer display-manager[1517]: /etc/vconsole.conf available Should I uncomment it or keep it commented? Or would it have any negative effects if uncommented?
Martin, I've been told that you might help to figure out why in this specific case setfont(8) fails to execute gzip. Sum-up: setfont(8) is spawned by a udev worker which is run from udev service which has MemoryDenyWriteExecute=yes. Basically this security option prevents services to execute data, please see man systemd.exec(5) for a full description of this option. It appears that this option has a negative side effect when setfont(8) tries to run "gzip -d ..." but I don't see why especially since this issue is only happening here and I couldn't reproduce it. Do you have an idea how this could be debugged further so we can understand why MemoryDenyWriteExecute=yes has a bad effect in this specific case ? Thanks.
(In reply to Franck Bui from comment #12) > Martin, I've been told that you might help to figure out why in this > specific case setfont(8) fails to execute gzip. I appreciate you consider me a person who can help you. > > Sum-up: setfont(8) is spawned by a udev worker which is run from udev > service which has MemoryDenyWriteExecute=yes. Basically this security option > prevents services to execute data, please see man systemd.exec(5) for a full > description of this option. > > It appears that this option has a negative side effect when setfont(8) tries > to run "gzip -d ..." but I don't see why especially since this issue is only > happening here and I couldn't reproduce it. I'm not much familiar with the systemd, but reading the documentation of the option: ``` ... Note that this option is incompatible with programs and libraries that generate program code dynamically at runtime, including JIT execution engines, executable stacks, and code "trampoline" feature of various C compilers. ... ``` I would suspect executable stacks that can occur for e.g. nested functions in C, but it's not the case of gzip: $ execstack /usr/bin/gzip - /usr/bin/gzip Or one can look at all sections of the binary and there are no X+W: $ readelf -S -W /usr/bin/gzip There are 30 section headers, starting at offset 0x13780: Section Headers: [Nr] Name Type Address Off Size ES Flg Lk Inf Al [ 0] NULL 0000000000000000 000000 000000 00 0 0 0 [ 1] .interp PROGBITS 00000000000002a8 0002a8 00001c 00 A 0 0 1 [ 2] .note.gnu.build-id NOTE 00000000000002c4 0002c4 000024 00 A 0 0 4 [ 3] .note.ABI-tag NOTE 00000000000002e8 0002e8 000020 00 A 0 0 4 [ 4] .hash HASH 0000000000000308 000308 000260 04 A 6 0 8 [ 5] .gnu.hash GNU_HASH 0000000000000568 000568 00003c 00 A 6 0 8 [ 6] .dynsym DYNSYM 00000000000005a8 0005a8 0007c8 18 A 7 1 8 [ 7] .dynstr STRTAB 0000000000000d70 000d70 00032f 00 A 0 0 1 [ 8] .gnu.version VERSYM 00000000000010a0 0010a0 0000a6 02 A 6 0 2 [ 9] .gnu.version_r VERNEED 0000000000001148 001148 000080 00 A 7 1 8 [10] .rela.dyn RELA 00000000000011c8 0011c8 000960 18 A 6 0 8 [11] .rela.plt RELA 0000000000001b28 001b28 0006c0 18 AI 6 25 8 [12] .init PROGBITS 0000000000003000 003000 000017 00 AX 0 0 4 [13] .plt PROGBITS 0000000000003020 003020 000490 10 AX 0 0 16 [14] .plt.got PROGBITS 00000000000034b0 0034b0 000008 08 AX 0 0 8 [15] .text PROGBITS 00000000000034c0 0034c0 00b221 00 AX 0 0 16 [16] .fini PROGBITS 000000000000e6e4 00e6e4 000009 00 AX 0 0 4 [17] .rodata PROGBITS 000000000000f000 00f000 0020c8 00 A 0 0 32 [18] .eh_frame_hdr PROGBITS 00000000000110c8 0110c8 000314 00 A 0 0 4 [19] .eh_frame PROGBITS 00000000000113e0 0113e0 001218 00 A 0 0 8 [20] .init_array INIT_ARRAY 00000000000138f0 0128f0 000008 08 WA 0 0 8 [21] .fini_array FINI_ARRAY 00000000000138f8 0128f8 000008 08 WA 0 0 8 [22] .data.rel.ro PROGBITS 0000000000013900 012900 0004e0 00 WA 0 0 32 [23] .dynamic DYNAMIC 0000000000013de0 012de0 0001f0 10 WA 7 0 8 [24] .got PROGBITS 0000000000013fd0 012fd0 000028 08 WA 0 0 8 [25] .got.plt PROGBITS 0000000000014000 013000 000258 08 WA 0 0 8 [26] .data PROGBITS 0000000000014260 013260 0003f0 00 WA 0 0 32 [27] .bss NOBITS 0000000000014660 013650 051000 00 WA 0 0 32 [28] .gnu_debuglink PROGBITS 0000000000000000 013650 000020 00 0 0 4 [29] .shstrtab STRTAB 0000000000000000 013670 00010a 00 0 0 1 Similarly for library of those the binary depends: $ ldd /usr/bin/gzip linux-vdso.so.1 (0x00007ffe39136000) libc.so.6 => /lib64/libc.so.6 (0x00007fc9ed3b0000) /lib64/ld-linux-x86-64.so.2 (0x00007fc9ed619000) > > Do you have an idea how this could be debugged further so we can understand > why MemoryDenyWriteExecute=yes has a bad effect in this specific case ? I would attach to the gzip with gdb and catch ta signal. Hope it will help. > > Thanks.
(In reply to Martin Liška from comment #13) > I would attach to the gzip with gdb and catch ta signal. > Hope it will help. Thanks for the feedback. Unfortunately, using gdb doesn't seem like an easy task because the issue only happens when gzip is run (indirectly) from a udev worker...
@Suse User, can you try to create a dumb service with MemoryDenyWriteExecute=yes which simply runs "gzip -d <compressed-file>" and see if you can reproduce your issue by starting the dumb service ? Something like: cat >/etc/systemd/system/debug-gzip.service<<EOF [Service] Type=oneshot RemainAfterExit=yes MemoryDenyWriteExecute=yes ExecStart=/usr/bin/gzip -d -k XXXXX EOF With "XXXXX" the path to a gzip compressed file. Thanks.
Franck Bui, I have replace XXXXX with /tmp/debug.zip and tried 'systemctl start debug-gzip.service'. The result (in 'journalctl -f') is: Aug 29 19:04:05 pc systemd[1]: Starting debug-gzip.service... Aug 29 19:04:05 pc gzip[2556]: /usr/bin/gzip: error while loading shared libraries: cannot make segment writable for relocation: Operation not permitted Aug 29 19:04:05 pc systemd[1]: debug-gzip.service: Main process exited, code=exited, status=127/n/a Aug 29 19:04:05 pc systemd[1]: debug-gzip.service: Failed with result 'exit-code'. Aug 29 19:04:05 pc systemd[1]: Failed to start debug-gzip.service.
Excellent you reproduced the issue. Can you try: > ExecStart=/usr/bin/strace -o /tmp/debug.strace /usr/bin/gzip -d -k /tmp/debug.zip and start again the debug service which should fail again but should also leave a file /tmp/debug.strace which will contain strace logs ? In that case, please attach /tmp/debug.strace.
Created attachment 816504 [details] debug.strace Attached.
Created attachment 816505 [details] Debug strace from Tumbleweed Hm, I can't reproduce that on my Tumbleweed on x86_64. Can you please provide information about your system? And please paste output of: $ execstack /usr/bin/gzip and $ readelf -S -W /usr/bin/gzip and $ ldd /usr/bin/gzip
Created attachment 816509 [details] requested info > Hm, I can't reproduce that on my Tumbleweed on x86_64. As per this ticket the report is for i586. > Can you please provide information about your system? The issue was noticed on Acer TravelMate 2410. Attaching the requested output too.
(In reply to Suse User from comment #20) > Created attachment 816509 [details] > requested info > > > Hm, I can't reproduce that on my Tumbleweed on x86_64. > > As per this ticket the report is for i586. Ah, ok, now I can reproduce that on i586 with: mprotect(0x435000, 53248, PROT_READ|PROT_WRITE|PROT_EXEC) = -1 EPERM (Operation not permitted) writev(2, [{iov_base="/usr/bin/gzip", iov_len=13}, {iov_base=": ", iov_len=2}, {iov_base="error while loading shared libra"..., iov_len=36}, {iov_base=": ", iov_len=2}, {iov_base="", iov_len=0}, {iov_base="", iov_len=0}, {iov_base="cannot make segment writable for"..., io v_len=43}, {iov_base=": ", iov_len=2}, {iov_base="Operation not permitted", iov_len=23}, {iov_base="\n", iov_len=1}], 10) = 122 It's unclear to me why the dynamic linker sets PROT_EXEC. @Andreas, @Micha: Can you help us here please?
Probably missing GNU_STACK header.
(In reply to Andreas Schwab from comment #22) > Probably missing GNU_STACK header. No, that's not the case: i586:/home/marxin # readelf -l -W /usr/bin/gzip | grep GNU_STACK GNU_STACK 0x000000 0x00000000 0x00000000 0x00000 0x00000 RW 0x10 i586:/home/marxin # readelf -l -W /lib/libc.so.6 | grep GNU_STACK GNU_STACK 0x000000 0x00000000 0x00000000 0x00000 0x00000 RW 0x10
It contains text relocations.
The assembler sources are not prepared for PIE.
(In reply to Andreas Schwab from comment #24) > It contains text relocations. Similarly to x86_64, which is fine with the MemoryDenyWriteExecute=yes ?
(In reply to Andreas Schwab from comment #25) > The assembler sources are not prepared for PIE. I don't understand that. Can you be please move descriptive?
If you want to use PIE you need to use position independent code.
(In reply to Andreas Schwab from comment #28) > If you want to use PIE you need to use position independent code. Ok, so is the issue invalid then?
Are you sure that gzip contains text relocation on x86-64? I can't imagine why it would work with MemoryDenyWriteExecute=yes then. In any case the issue is valid, and there are only two solutions: either gzip needs to be patched so that the containing assembler sequences don't create position dependend code (which are what ultimately cause .text relocations), or gzip must not be called under the control of MemoryDenyWriteExecute=yes on i586. That setting is incompatible with any executable that isn't position independend. I'd say using it for any service that potentially calls other executables than just the main service exe (like here, some executable actually calling setfont, calling gzip) is a bad idea.
This is about i586.
Yes, I didn't say different in comment #30. I was reacting to Martins claim of comment #26, but that was only a side remark. The main thing still is that on i586 either gzip needs to be changed or MemoryDenyWriteExecute=yes be disabled.
(In reply to Michael Matz from comment #30) > Are you sure that gzip contains text relocation on x86-64? I can't imagine > why it would work with MemoryDenyWriteExecute=yes then. Martin, just to be sure, can you please answer to Michael ? Thanks.
(In reply to Franck Bui from comment #33) > (In reply to Michael Matz from comment #30) > > Are you sure that gzip contains text relocation on x86-64? I can't imagine > > why it would work with MemoryDenyWriteExecute=yes then. > > Martin, just to be sure, can you please answer to Michael ? > > Thanks. I discussed that with Michael offline and the problem is in gzip program. The way it's written leads to the need of executable and writable section.
(In reply to Martin Liška from comment #34) > I discussed that with Michael offline and the problem is in gzip program. > The way it's written leads to the need of executable and writable section. But that doesn't explain why x86_64 works with "MemoryDenyWriteExecute=yes" or do you mean the way gzip is written for i586 only ? Do you have any pointer ? Thanks.
Can you Michael please summarize the reasons why it does not work on i586 and work on x86_64? Thanks.
http://git.savannah.gnu.org/cgit/gzip.git/tree/lib/match.c#n52
(Let's please not distract ourselves too much with the x86_64 sub-thread here, it should be clear what the problem is on i586 meanwhile. But Martin: you claimed that the x86_64 binary also contains text relocations and still works with DenyWriteExecute=yes (comment #26), I merely asked back why you think so)
(In reply to Michael Matz from comment #38) > (Let's please not distract ourselves too much with the x86_64 sub-thread > here, > it should be clear what the problem is on i586 meanwhile. But Martin: you > claimed that the x86_64 binary also contains text relocations and still > works with DenyWriteExecute=yes (comment #26), I merely asked back why you > think so) I tested that locally on my machine with DenyWriteExecute=yes and the service defined in https://bugzilla.suse.com/show_bug.cgi?id=1143125#c15.
(In reply to Martin Liška from comment #39) > (In reply to Michael Matz from comment #38) > > (Let's please not distract ourselves too much with the x86_64 sub-thread > > here, > > it should be clear what the problem is on i586 meanwhile. But Martin: you > > claimed that the x86_64 binary also contains text relocations and still > > works with DenyWriteExecute=yes (comment #26), I merely asked back why you > > think so) > > I tested that locally on my machine with DenyWriteExecute=yes and the > service defined in https://bugzilla.suse.com/show_bug.cgi?id=1143125#c15. I was asking why you think it contains text relocations.
> > I was asking why you think it contains text relocations. > Yes. I claimed that, but wrongly :)
Created attachment 850365 [details] full journal + systemd.log_level=debug Hello, something similar happens in JeOS images as well. Eventually, systemd-vconsole-setup does its job, but it yields warnings messages earlier during boot. Currently, in QA we are trying to track messages starting with prio warning, so it does not create any confusion after release. Is there anything what can be done over here ? Does it make sense to set back "RemainAfterExit=no" as it was modified with https://github.com/systemd/systemd/issues/9568 ?
(In reply to Martin Loviska from comment #42) > something similar happens in JeOS images as well. Eventually, > systemd-vconsole-setup does its job, but it yields warnings messages earlier > during boot. Assuming that you were referring to the following message: [ 11.157594] localhost systemd[1]: systemd-vconsole-setup.service: Main process exited, code=killed, status=15/TERM the problem seems different, especially since your system is x86_64. Can you please open another bug report and attach the debug logs when your problem is reproduced ? The logs you attached doesn't have the debug messages when the system booted the first time. They were only activated during the second boot but I couldn't find any problem with vconsole-setup in this part.
(In reply to Franck Bui from comment #43) > (In reply to Martin Loviska from comment #42) > > something similar happens in JeOS images as well. Eventually, > > systemd-vconsole-setup does its job, but it yields warnings messages earlier > > during boot. > > Assuming that you were referring to the following message: > > [ 11.157594] localhost systemd[1]: systemd-vconsole-setup.service: Main > process exited, code=killed, status=15/TERM > > the problem seems different, especially since your system is x86_64. > > Can you please open another bug report and attach the debug logs when your > problem is reproduced ? > > The logs you attached doesn't have the debug messages when the system booted > the first time. They were only activated during the second boot but I > couldn't find any problem with vconsole-setup in this part. Oh, yes you are right, sorry for this inconvenience! Should be fixed in the specific bug report attached below. Thanks! Let's continue with https://bugzilla.suse.com/show_bug.cgi?id=1187618
I finally submitted https://build.opensuse.org/request/show/903071, which should disable the use of assembler code (specific to i586 arch). According to Fedora (who turned it off several years ago), the assembler code was slower anyway. Hence closing. Thanks everyone.