Bugzilla – Bug 1172846
gcc10 regression: C++ nondeterministic debuginfo generation
Last modified: 2021-03-02 15:10:18 UTC
Since we switched to gcc10 as default compiler last week, several formerly reproducible packages have changed status to unreproducible in my reproducible-builds tests for openSUSE. Examples are dustrac libzypp rosegarden solarus spirv-tools watchman z3 When building without debuginfo, objdump -s only shows a diff in the .note.gnu.build-id section But when building with debuginfo, many more diffs appear in debug files. All these builds pull in gcc10-c++-10.1.1 so the problem might be limited to C++ .
Did you check whether it reproduces when not using -flto? Which package is the fastest to build locally?
OK, reproduced with dustrac which builds in about 200s for me where with just two builds dustrac-editor ends up with the same build-id but the one for dustrac-game differs. Trying with %define _lto_cflags %{nil} now.
Again with a sample of two without -flto the build-ids are the same. Things to check: - does ld when using the linker plugin use any of the LTO IL containing files for computing the build-id (LTO IL was in the past "stable", that may have changed) - does debug info copying somehow copy / genrate data that is taken into account and that differs - verify the generated code really does not differ but not today.
dustrac is among the fastest to build. watchman is even a bit faster. I also discovered strawberry (slower to build: around 500s) that becomes reproducible when I disable lto but also becomes reproducible when I replace /dev/*random with /dev/zero via osc build -x reproducible-faketools-random --vm-type=kvm
I'm taking look at dustrac: --- /home/marxin/Programming/DustRacing2D/objdir/1 +++ /home/marxin/Programming/DustRacing2D/objdir/3 ├── readelf --wide --sections {} │ @@ -31,17 +31,17 @@ │ [26] .got.plt PROGBITS 0000000000529000 128000 000e88 08 WA 0 0 8 │ [27] .data PROGBITS 0000000000529ea0 128ea0 000088 00 WA 0 0 32 │ [28] .bss NOBITS 0000000000529f40 128f28 000d60 00 WA 0 0 64 │ [29] .comment PROGBITS 0000000000000000 128f28 000056 01 MS 0 0 1 │ [30] .debug_aranges PROGBITS 0000000000000000 128f80 0006a0 00 0 0 16 │ [31] .debug_info PROGBITS 0000000000000000 129620 16e531e 00 0 0 1 │ [32] .debug_abbrev PROGBITS 0000000000000000 180e93e 082131 00 0 0 1 │ - [33] .debug_line PROGBITS 0000000000000000 1890a6f 15964f 00 0 0 1 │ - [34] .debug_str PROGBITS 0000000000000000 19ea0be 441e22 01 MS 0 0 1 │ - [35] .debug_loc PROGBITS 0000000000000000 1e2bee0 50bed8 00 0 0 1 │ + [33] .debug_line PROGBITS 0000000000000000 1890a6f 15964b 00 0 0 1 │ + [34] .debug_str PROGBITS 0000000000000000 19ea0ba 441e22 01 MS 0 0 1 │ + [35] .debug_loc PROGBITS 0000000000000000 1e2bedc 50bed8 00 0 0 1 │ [36] .debug_ranges PROGBITS 0000000000000000 2337dc0 170c40 00 0 0 16 │ [37] .symtab SYMTAB 0000000000000000 24a8a00 058ea8 18 38 12169 8 │ [38] .strtab STRTAB 0000000000000000 25018a8 03eec9 00 0 0 1 │ [39] .shstrtab STRTAB 0000000000000000 2540771 000186 00 0 0 1 │ Key to Flags: │ W (write), A (alloc), X (execute), M (merge), S (strings), I (info), │ L (link order), O (extra OS processing required), G (group), T (TLS), ├── readelf --wide --notes {} │ @@ -1,8 +1,8 @@ │ │ Displaying notes found in: .note.gnu.build-id │ Owner Data size Description │ - GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring) Build ID: 2202f33b89481f11f112a8de9d1151d73f98aacf │ + GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring) Build ID: 5bd81d5b3554cdcd158d6ea39ce0d950544821b0 │ │ Displaying notes found in: .note.ABI-tag │ Owner Data size Description │ GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag) OS: Linux, ABI: 3.2.0 ├── readelf --wide --debug-dump=rawline {} │ @@ -362136,15 +362136,15 @@ │ [0x0009cfe9] Special opcode 6: advance Address by 0 to 0x4a55c0 and Line by 1 to 421 (view 5) │ [0x0009cfea] Special opcode 61: advance Address by 4 to 0x4a55c4 and Line by 0 to 421 │ [0x0009cfeb] Set File Name to entry 36 in the File Name Table │ [0x0009cfed] Set column to 5 │ [0x0009cfef] Set is_stmt to 1 │ [0x0009cff0] Advance Line by 1648 to 2069 │ [0x0009cff3] Copy (view 1) │ - [0x0009cff4] Set column to 7 │ + [0x0009cff4] Set column to 3 │ [0x0009cff6] Set is_stmt to 0 │ [0x0009cff7] Special opcode 7: advance Address by 0 to 0x4a55c4 and Line by 2 to 2071 (view 2) │ [0x0009cff8] Set File Name to entry 22 in the File Name Table │ [0x0009cffa] Set column to 5 │ [0x0009cffc] Advance Line by -1988 to 83 │ [0x0009cfff] Special opcode 131: advance Address by 9 to 0x4a55cd and Line by 0 to 83 │ [0x0009d000] Special opcode 159: advance Address by 11 to 0x4a55d8 and Line by 0 to 83 │ @@ -362201,15 +362201,15 @@ │ [0x0009d054] Copy (view 1) │ [0x0009d055] Set File Name to entry 14 in the File Name Table │ [0x0009d057] Advance Line by 12 to 162 │ [0x0009d059] Copy (view 2) │ [0x0009d05a] Set is_stmt to 0 │ [0x0009d05b] Copy (view 3) │ [0x0009d05c] Set File Name to entry 5 in the File Name Table │ - [0x0009d05e] Set column to 7 │ + [0x0009d05e] Set column to 3 │ [0x0009d060] Advance Line by -29 to 133 │ [0x0009d062] Copy (view 4) │ [0x0009d063] Special opcode 187: advance Address by 13 to 0x4a55ff and Line by 0 to 133 │ [0x0009d064] Set File Name to entry 36 in the File Name Table │ [0x0009d066] Advance Line by 1938 to 2071 │ [0x0009d069] Copy (view 1) │ [0x0009d06a] Special opcode 131: advance Address by 9 to 0x4a5608 and Line by 0 to 2071 ...
Note the most interesting part is why the build id changes - the build id does not factor in .debug_info sections (just allocated sections). Yes, it would be nice if debug info would not change but that's secondary.
OK, so doing a LTO plugin link of #include <iostream> int main() { std::cout << "Hello world!" << std::endl; return 0; } produces a different build-id compared to linking the LTRANS object and the early debug info object (as preserved by -save-temps). The build-id also depends on the order of the early debug object and the LTRANS object on the linker command line (but IMHO the early debug object should not contribute anything to the build-id). Maybe ld takes into account relocation sections and symtab. That said, my theory is that the build-id is computed from the objects on the LTO link line (maybe not exclusively) and that the LTO IL became unstable. It does generate_build_id (abfd, style, bed->s->checksum_contents, id_bits, size); which eventually calls bfd_checksum_contents in elfcode.h. That processes the elf header for all objects of the BFD, the program headers and section headers and all sections contents (contrary to my expectations...). Eventually plugin taken objects could be skipped (I think that makes sense), but then the differences in debug info would still make the build id different. Indeed a short experiment shows that the build-id is different -g vs. -g0.
For the debug info seen above it's probably interesting to do the link with -save-temps so that we can see whether the early generated debug differs (I would suspect so - IIRC we drop columns for late compilation) or the debug form the ltrans objects.
(In reply to Richard Biener from comment #7) > OK, so doing a LTO plugin link of > > #include <iostream> > int main() { > std::cout << "Hello world!" << std::endl; > return 0; > } How exactly do you link this in order to get a different buildid? I can bisect that probably..
(In reply to Martin Liška from comment #9) > (In reply to Richard Biener from comment #7) > > OK, so doing a LTO plugin link of > > > > #include <iostream> > > int main() { > > std::cout << "Hello world!" << std::endl; > > return 0; > > } > > How exactly do you link this in order to get a different buildid? I can > bisect that probably.. I bet it's always been that way, so I think the best way forward is to get a handle on the debuginfo differences. If they show for early debug then that should be easy to debug (single source TU).
I was able to bisect the '--save-temps' vs. '' and it goes to: r8-4047-g8008dd1c93cf12b8(23 Oct 2017 16:10)(jakub@redhat.com): [took: 1.207s] result: FAILED (1) --- 1 2020-06-16 10:45:32.267974214 +0200 +++ 2 2020-06-16 10:45:32.675971887 +0200 @@ -1,7 +1,7 @@ Displaying notes found in: .note.gnu.build-id Owner Data size Description - GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring) Build ID: d025e03644687e75b03b2a4774388b72d1661a6a + GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring) Build ID: 2181bd96afc787ca9bc66a65d93828c8cc193ea9 Displaying notes found in: .note.ABI-tag Owner Data size Description common.opt (gcolumn-info): Enable by default. * common.opt (gcolumn-info): Enable by default. * doc/invoke.texi (gcolumn-info): Document new default. * lib/scanasm.exp (dg-function-on-line): Accept optional column info. * gcc.dg/debug/dwarf2/pr53948.c: Likewise. * g++.dg/debug/dwarf2/pr77363.C: Likewise. * gcc.dg/debug/dwarf2/asm-line1.c: Add -gno-column-info to dg-options. * gcc.dg/debug/dwarf2/discriminator.c: Likewise. * g++.dg/debug/dwarf2/typedef6.C: Likewise. From-SVN: r254010 So I bet there's a mess in column information? It was also printed here: ├── readelf --wide --debug-dump=rawline {} │ @@ -362136,15 +362136,15 @@ │ [0x0009cfe9] Special opcode 6: advance Address by 0 to 0x4a55c0 and Line by 1 to 421 (view 5) │ [0x0009cfea] Special opcode 61: advance Address by 4 to 0x4a55c4 and Line by 0 to 421 │ [0x0009cfeb] Set File Name to entry 36 in the File Name Table │ [0x0009cfed] Set column to 5 │ [0x0009cfef] Set is_stmt to 1 │ [0x0009cff0] Advance Line by 1648 to 2069 │ [0x0009cff3] Copy (view 1) │ - [0x0009cff4] Set column to 7 │ + [0x0009cff4] Set column to 3
Note my experiment wasn't about -save-temps and '' but about whether LTO IL is possibly leaking into .build-id, trying to compare a link with LTO IL objects and plugin invocation with a link of the final objects. But there's more differences in the resulting ELF file so that experiment did not prove anything about the LTO IL contribution or not (and I used -save-temps consistently)
I probably have it: LTO sections differ in between runs, we use a random identifier: - [ 9] .gnu.lto_.profile.93500ee746ee356 PROGBITS 0000000000000000 000207 00000f 00 E 0 0 1 - [10] .gnu.lto_.icf.93500ee746ee356 PROGBITS 0000000000000000 000216 000011 00 E 0 0 1 - [11] .gnu.lto_.jmpfuncs.93500ee746ee356 PROGBITS 0000000000000000 000227 000011 00 E 0 0 1 vs. + [ 9] .gnu.lto_.profile.5b9384e6c0728601 PROGBITS 0000000000000000 000207 00000f 00 E 0 0 1 + [10] .gnu.lto_.icf.5b9384e6c0728601 PROGBITS 0000000000000000 000216 000011 00 E 0 0 1 + [11] .gnu.lto_.jmpfuncs.5b9384e6c0728601 PROGBITS 0000000000000000 000227 000011 00 E 0 0 1 but the thing is that the names go to .shstrtab and I see there a different in size: - [26] .shstrtab STRTAB 0000000000000000 0006d0 000256 00 0 0 1 + [26] .shstrtab STRTAB 0000000000000000 0006d0 000262 00 0 0 1 and - Start of section headers: 2344 (bytes into file) + Start of section headers: 2360 (bytes into file) Am I right that the strings are merged in the section and so that we can end with a different section size?
(In reply to Martin Liška from comment #13) > I probably have it: LTO sections differ in between runs, we use a random > identifier: > > - [ 9] .gnu.lto_.profile.93500ee746ee356 PROGBITS 0000000000000000 > 000207 00000f 00 E 0 0 1 > - [10] .gnu.lto_.icf.93500ee746ee356 PROGBITS 0000000000000000 > 000216 000011 00 E 0 0 1 > - [11] .gnu.lto_.jmpfuncs.93500ee746ee356 PROGBITS 0000000000000000 > 000227 000011 00 E 0 0 1 > > vs. > > + [ 9] .gnu.lto_.profile.5b9384e6c0728601 PROGBITS 0000000000000000 > 000207 00000f 00 E 0 0 1 > + [10] .gnu.lto_.icf.5b9384e6c0728601 PROGBITS 0000000000000000 > 000216 000011 00 E 0 0 1 > + [11] .gnu.lto_.jmpfuncs.5b9384e6c0728601 PROGBITS 0000000000000000 > 000227 000011 00 E 0 0 1 > > but the thing is that the names go to .shstrtab and I see there a different > in size: > - [26] .shstrtab STRTAB 0000000000000000 0006d0 000256 00 > 0 0 1 > + [26] .shstrtab STRTAB 0000000000000000 0006d0 000262 00 > 0 0 1 > > and > > - Start of section headers: 2344 (bytes into file) > + Start of section headers: 2360 (bytes into file) > > Am I right that the strings are merged in the section and so that we can end > with a different section size? I'm not sure - as far as I read the BFD code it iterates over _output_ sections. So what's in the output section matters. Yes, if those section names somehow prevail that would be an issue. But as said the difference in .debug_line of the resulting object will be enough to trigger a build-id difference (and Bernhard said without debug info the differences vanish).
> I'm not sure - as far as I read the BFD code it iterates over _output_ > sections. So what's in the output section matters. Yes, if those > section names somehow prevail that would be an issue. Ah, ok. It's as simple as we generate different length of random section suffixes: profile.57fa86a8d74b94c6 profile.b9682d2e48eea9e > But as said the > difference in .debug_line of the resulting object will be enough to > trigger a build-id difference (and Bernhard said without debug info > the differences vanish).
> I'm not sure - as far as I read the BFD code it iterates over _output_ > sections. So what's in the output section matters. You are right, I fixed length of the random seed and it does not help. > Yes, if those > section names somehow prevail that would be an issue. But as said the > difference in .debug_line of the resulting object will be enough to > trigger a build-id difference (and Bernhard said without debug info > the differences vanish). Yes, without debug info, buildid is stable. I see a divergence in a debug info column of an output ltrans file. Dunno how to carry..
So what's interesting is that the divergence is always from BLOCK_SOURCE_LOCATION of BLOCKs representing calls, thus in DW_TAG_inlined_subroutine the DW_AT_call_column value. And within a single ltrans unit it's always a difference of two values like columns 13 changing to 3. Other values are 7 and 15 or 7 and 16. I also observe more than the above like differing line number programs: [0x0000aa40] Set File Name to entry 11 in the File Name Ta [0x0000aa40] Set File Name to entry 11 in the File Name Ta [0x0000aa42] Set is_stmt to 0 | [0x0000aa42] Set column to 16 [0x0000aa43] Advance Line by -1525 to 160 | [0x0000aa44] Set is_stmt to 0 [0x0000aa46] Copy (view 3) | [0x0000aa45] Advance Line by -1525 to 160 [0x0000aa47] Special opcode 61: advance Address by 4 to 0x | [0x0000aa48] Copy (view 3) [0x0000aa48] Advance Line by 27 to 187 | [0x0000aa49] Special opcode 61: advance Address by 4 to 0x [0x0000aa4a] Copy (view 1) | [0x0000aa4a] Advance Line by 27 to 187 [0x0000aa4b] Special opcode 75: advance Address by 5 to 0x | [0x0000aa4c] Copy (view 1) there's a "missing" Set column, probably fallout of the other change since the preceeding column was 7 (this is the 7 to 16 change). Further ripple-down effects exist. I'm not sure how to get a handle on the column number changes, all early debug objects compare equal besides offsets in archive files not matching so I didn't check those. Likely the usual sorting problem. In another case there's columns dropped: <5><267bf>: Abbrev Number: 30 (DW_TAG_inlined_subroutine) | <5><267bf>: Abbrev Number: 68 (DW_TAG_inlined_subroutine) <267c0> DW_AT_abstract_origin: <0x56025> <267c0> DW_AT_abstract_origin: <0x56025> <267c4> DW_AT_ranges : 0xb170 <267c4> DW_AT_ranges : 0xb170 <267c8> DW_AT_call_file : 14 <267c8> DW_AT_call_file : 14 <267c9> DW_AT_call_line : 2209 <267c9> DW_AT_call_line : 2209 <267cb> DW_AT_call_column : 38 | <267cb> DW_AT_sibling : <0x2691d> <267cc> DW_AT_sibling : <0x26920> | <6><267cf>: Abbrev Number: 7 (DW_TAG_formal_parameter) with of course more ripple-down effects. When I reduce the minimal LTRANS size I get more LTRANS units but the issue prevails (interestingly only in dustrac-game again, not dustrac-editor). The actual column changes are different though.
Re-linking the executable from the same IL files produces consistent results. The IL files differ because of the difference in the section hash (so diffs are hard to come by). And with a sample of three builds, supplying -frandom-seed=1 makes the results equal (but that's not a solution since it will break C++ anonymous namespace uniqueness).
(In reply to Richard Biener from comment #18) > Re-linking the executable from the same IL files produces consistent results. > > The IL files differ because of the difference in the section hash (so diffs > are hard to come by). > > And with a sample of three builds, supplying -frandom-seed=1 makes the > results equal (but that's not a solution since it will break C++ anonymous > namespace uniqueness). Can't that be related to e.g. different type merging which will be influenced by the order of LTO sections (that are keyed with the random numbers)?
(In reply to Martin Liška from comment #19) > (In reply to Richard Biener from comment #18) > > Re-linking the executable from the same IL files produces consistent results. > > > > The IL files differ because of the difference in the section hash (so diffs > > are hard to come by). > > > > And with a sample of three builds, supplying -frandom-seed=1 makes the > > results equal (but that's not a solution since it will break C++ anonymous > > namespace uniqueness). > > Can't that be related to e.g. different type merging which will be > influenced by the order of LTO sections (that are keyed with the random > numbers)? Might be, but I think the order we merge depends not on the order of sections in the file, does it? And the order of section streaming doesn't depend on the name but the order we create them.
Is there any fix in sight? This bug is annoying as it affects many packages and might make me miss unreproducibility from other sources.
(In reply to Bernhard Wiedemann from comment #21) > Is there any fix in sight? > This bug is annoying as it affects many packages and might make me miss > unreproducibility from other sources. I hoped it would go away magically so I assume it didn't. So I'll try to investigate some more this week - sorry for the delay.
OK, so I traced this to 2019-10-23 Jan Hubicka <hubicka@ucw.cz> * lto-streamer-out.c (output_constructor): Push CTORS_OUT timevar. (cmp_symbol_files): New. (lto_output): Copy sections in file order. * lto-streamer.h (lto_file_decl_data): Add field order. where cmp_symbol_files does /* Order within static library. */ if (n1->lto_file_data && n1->lto_file_data->id != n2->lto_file_data->id) { if (n1->lto_file_data->id > n2->lto_file_data->id) return 1; if (n1->lto_file_data->id < n2->lto_file_data->id) return -1; } but lto_file_data->id is the random hash we suffix LTO sections with. Thinking about a fix.
Created attachment 839372 [details] patch for gcc10 Here's the patch, with it dustrac produces the same binaries (verified 5 times). This is going through upstream and staging for GCC 10 so in case you want to give this local testing you can pick up this patch temporarily.
Thanks. I adopted the patch into my overlay repo https://build.opensuse.org/package/show/home:bmwiedemann:reproducible/gcc10 This patch fixed all those newly unreproducible C++ builds. I will also notice if some builds or testsuites suddenly break. If wanted, I can also throw newer patch revisions in there to give it some coverage before it hits Factory.
The identical fix has hit upstream and I updated devel:gcc/gcc10 with it. Since there will be a GCC 10.2 release candidate later this week which I'd like to push to Factory I'll defer pushing to Factory after pulling the RC.
This is an autogenerated message for OBS integration: This bug (1172846) was mentioned in https://build.opensuse.org/request/show/821094 Factory / gcc10
This is an autogenerated message for OBS integration: This bug (1172846) was mentioned in https://build.opensuse.org/request/show/822560 Factory / gcc10
This is an autogenerated message for OBS integration: This bug (1172846) was mentioned in https://build.opensuse.org/request/show/823312 Factory / gcc10
SUSE-SU-2020:2947-1: An update that solves one vulnerability, contains two features and has 5 fixes is now available. Category: security (moderate) Bug References: 1172798,1172846,1173972,1174753,1174817,1175168 CVE References: CVE-2020-13844 JIRA References: ECO-2373,SLE-12297 Sources used: SUSE Linux Enterprise Server for SAP 15 (src): cross-nvptx-gcc10-10.2.1+git583-1.3.2, gcc10-10.2.1+git583-1.3.4, nvptx-tools-1.0-4.3.2 SUSE Linux Enterprise Server 15-LTSS (src): gcc10-10.2.1+git583-1.3.4 SUSE Linux Enterprise Module for Development Tools 15-SP2 (src): cross-nvptx-gcc10-10.2.1+git583-1.3.2, gcc10-10.2.1+git583-1.3.4, nvptx-tools-1.0-4.3.2 SUSE Linux Enterprise Module for Development Tools 15-SP1 (src): cross-nvptx-gcc10-10.2.1+git583-1.3.2, gcc10-10.2.1+git583-1.3.4, nvptx-tools-1.0-4.3.2 SUSE Linux Enterprise Module for Basesystem 15-SP2 (src): gcc10-10.2.1+git583-1.3.4 SUSE Linux Enterprise Module for Basesystem 15-SP1 (src): gcc10-10.2.1+git583-1.3.4 SUSE Linux Enterprise High Performance Computing 15-LTSS (src): cross-nvptx-gcc10-10.2.1+git583-1.3.2, gcc10-10.2.1+git583-1.3.4, nvptx-tools-1.0-4.3.2 SUSE Linux Enterprise High Performance Computing 15-ESPOS (src): cross-nvptx-gcc10-10.2.1+git583-1.3.2, gcc10-10.2.1+git583-1.3.4, nvptx-tools-1.0-4.3.2 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
openSUSE-SU-2020:1693-1: An update that solves one vulnerability and has 5 fixes is now available. Category: security (moderate) Bug References: 1172798,1172846,1173972,1174753,1174817,1175168 CVE References: CVE-2020-13844 JIRA References: Sources used: openSUSE Leap 15.2 (src): cross-nvptx-gcc10-10.2.1+git583-lp152.2.1, gcc10-10.2.1+git583-lp152.2.2, nvptx-tools-1.0-lp152.4.3.2
openSUSE-SU-2020:1692-1: An update that solves one vulnerability and has 5 fixes is now available. Category: security (moderate) Bug References: 1172798,1172846,1173972,1174753,1174817,1175168 CVE References: CVE-2020-13844 JIRA References: Sources used: openSUSE Leap 15.1 (src): cross-nvptx-gcc10-10.2.1+git583-lp151.2.1, gcc10-10.2.1+git583-lp151.2.2, nvptx-tools-1.0-lp151.3.3.2
SUSE-SU-2020:3263-1: An update that solves one vulnerability, contains two features and has 5 fixes is now available. Category: security (moderate) Bug References: 1172798,1172846,1173972,1174753,1174817,1175168 CVE References: CVE-2020-13844 JIRA References: ECO-2373,SLE-12297 Sources used: SUSE OpenStack Cloud Crowbar 9 (src): gcc10-10.2.1+git583-1.3.5 SUSE OpenStack Cloud Crowbar 8 (src): gcc10-10.2.1+git583-1.3.5 SUSE OpenStack Cloud 9 (src): gcc10-10.2.1+git583-1.3.5 SUSE OpenStack Cloud 8 (src): gcc10-10.2.1+git583-1.3.5 SUSE OpenStack Cloud 7 (src): gcc10-10.2.1+git583-1.3.5 SUSE Linux Enterprise Server for SAP 12-SP4 (src): gcc10-10.2.1+git583-1.3.5 SUSE Linux Enterprise Server for SAP 12-SP3 (src): gcc10-10.2.1+git583-1.3.5 SUSE Linux Enterprise Server for SAP 12-SP2 (src): gcc10-10.2.1+git583-1.3.5 SUSE Linux Enterprise Server 12-SP5 (src): gcc10-10.2.1+git583-1.3.5 SUSE Linux Enterprise Server 12-SP4-LTSS (src): gcc10-10.2.1+git583-1.3.5 SUSE Linux Enterprise Server 12-SP3-LTSS (src): gcc10-10.2.1+git583-1.3.5 SUSE Linux Enterprise Server 12-SP3-BCL (src): gcc10-10.2.1+git583-1.3.5 SUSE Linux Enterprise Server 12-SP2-LTSS (src): gcc10-10.2.1+git583-1.3.5 SUSE Linux Enterprise Server 12-SP2-BCL (src): gcc10-10.2.1+git583-1.3.5 SUSE Linux Enterprise Module for Toolchain 12 (src): cross-nvptx-gcc10-10.2.1+git583-1.3.1, gcc10-10.2.1+git583-1.3.5 SUSE Enterprise Storage 5 (src): gcc10-10.2.1+git583-1.3.5 HPE Helion Openstack 8 (src): gcc10-10.2.1+git583-1.3.5 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Should be fixed now.