Judging from https://github.com/rust-lang/rust/blob/1.55.0/.gitmodules the bundled LLVM seems to be LLVM 12. Maybe rust 1.55 should use llvm12 instead of llvm = llvm13 for now?

Rust 1.56.0 seems to be using LLVM 13 at upstream, and looking through [1] I don't see any commits that look like they fix a miscompilation. The patches for Rust 1.55 on top of LLVM 12 [2] look mostly inconspicuous as well. Maybe [3] could be relevant, but likely not.

[1] https://github.com/llvm/llvm-project/compare/llvmorg-13.0.0...rust-lang:rustc/13.0-2021-08-08
[2] https://github.com/llvm/llvm-project/compare/llvmorg-12.0.1...rust-lang:rustc/12.0-2021-07-10
[3] https://github.com/llvm/llvm-project/commit/c40829eebaa018a4bd96f573de0a9cd2bf34524e

Currently I'm working to resolve some issues with rust1.55 that have appeared in OpenSUSE (unrelated to this). I'm okay to enabled the bundled LLVM for SLE if that would immediately resolve this issue for you. Which arches do you need this enabled for?

(In reply to William Brown from comment #2)
> I'm okay to enabled the bundled
> LLVM for SLE if that would immediately resolve this issue for you. Which
> arches do you need this enabled for?

Please note that this issue is about Tumbleweed explicitely and not SLE.
And which archs is a good question. I assume at least all supported by TW.

(In reply to Aaron Puchert from comment #1)
> Judging from https://github.com/rust-lang/rust/blob/1.55.0/.gitmodules the
> bundled LLVM seems to be LLVM 12. Maybe rust 1.55 should use llvm12 instead
> of llvm = llvm13 for now?

I like this proposal! In fact I have patched my rust1.55 package now to BuildRequire: llvm12-devel; the rust packages are ready to be used for TW/x86_64 and I'm currently building firefox93 with these new packages. Will report how that will work when finished!

Update: I've now rebuilt firefox93 locally using "osc build ..." with packages downloaded from OBS. The resulting MozillaFirefox packages work as expected, not any glitches.

Summary: adding an explicit "BuildRequires: llvm12-devel" for Tumbleweed helps to build a functioning Rust compiler for Tumbleweed.

Details available at:

<https://build.opensuse.org/package/show/home:manfred-h:devel:languages:rust:rust-1.55/rust1.55>

FWIW, I created the following SR for this now:

  <https://build.opensuse.org/request/show/927971>

Another update: I've now also successfully built and tested firefox91 (ESR)!

Details for both firefox versions:

* <https://build.opensuse.org/package/show/home:manfred-h:mozilla/firefox93>
* <https://build.opensuse.org/package/show/home:manfred-h:mozilla:esr/firefox91>

Thanks Manfred, sorry for the delays in responding. I've accepted the SR and I'll get the fixes for llvm bundling through to SLE as well.

Sorry for being slightly OT.

Do we have a way to find out, which packages were compiled with llvm13?

I even see some issues with Mesa (and VNC: https://github.com/TigerVNC/tigervnc/issues/1220), which might be another llvm13 victim in the end?

I'm not THE expert but I don't think that LLVM13 is broken in itself. It seems to be the combination of rust 1.55 and llvm13.

So I would check for packages built using rust.
Actually I don't know how to check easily but in theory it's packages with
BuildRequires: rust*
to check for.

(In reply to Hans-Peter Jansen from comment #12)
> Do we have a way to find out, which packages were compiled with llvm13?
Yes:

    osc whatdependson openSUSE:Factory llvm13 standard x86_64

(In reply to Wolfgang Rosenauer from comment #13)
> I'm not THE expert but I don't think that LLVM13 is broken in itself. It
> seems to be the combination of rust 1.55 and llvm13.
I'm not aware of any issues in llvm13, and Rust doesn't seem to have any relevant patches on their bundled LLVM 13 in Rust 1.56. So I presume there was just a misunderstanding between LLVM 13 and Rust 1.55 which has been cleared up.

> So I would check for packages built using rust.
> Actually I don't know how to check easily but in theory it's packages with
> BuildRequires: rust*
> to check for.
Same procedure:

    osc whatdependson openSUSE:Factory rust1.55 standard x86_64

(In reply to Manfred Hollstein from comment #0)
>   <https://build.opensuse.org/project/show/home:manfred-h:mozilla>

... seems to work for me.

Thanks guys, this is one of the most helpful bug records ever.

@Manfred: I depend so much on Firefox for my daily work (the company driven with 1000 portals), that I've postponed any tests to the weekend, where I plan to test your builds. Thank you very much for providing them (including ESR builds). Because of https://bugzilla.mozilla.org/show_bug.cgi?id=1716989, I'm still on 91, but your builds might even make a difference in this respect as well, therefor I will gradually upgrade to your 91 build and (after another day with backup) to 94.

@Aaron, @William, @Wolfgang: one thing, that makes me nervous is the errors behavior. In Firefox, it seems to be sandbox related, but no real failure there, apart from some hints of failing access to /dev/random. Why/how does this break specific areas like WebRTC only, while most of the rest is acting fine?

Without understanding, what and how it is going wrong, we'll have a hard time to look for similar issues in other packages.

If it's, after all, "just" an "impedance mismatch" between llvm 13 and rust 1.55, something in the interface logic between them is broken, and the lesson learned from this should be at least: build rust only with the llvm, that upstream uses.

(In reply to Hans-Peter Jansen from comment #16)

> If it's, after all, "just" an "impedance mismatch" between llvm 13 and rust
> 1.55, something in the interface logic between them is broken, and the
> lesson learned from this should be at least: build rust only with the llvm,
> that upstream uses.

I made this suggestion in the PR that fixed the issue, but they make me to understand that:

1. llvm is very slow to compile, so when compiling rustc / cargo is good to use the llvm from the system to gain some speed.

2. rustc is expected to work with some range of llvm versions

I think that they are fair points, and TW can help in detecting early issues.

But still I think that there should be an outcome of this bug, like working on enabling the tests inside the %check section on the rust package.

(In reply to Hans-Peter Jansen from comment #16)
> Thanks guys, this is one of the most helpful bug records ever.

You are most welcome! We always try to help here :) 

> 
> @Aaron, @William, @Wolfgang: one thing, that makes me nervous is the errors
> behavior. In Firefox, it seems to be sandbox related, but no real failure
> there, apart from some hints of failing access to /dev/random. Why/how does
> this break specific areas like WebRTC only, while most of the rest is acting
> fine?
> 
> Without understanding, what and how it is going wrong, we'll have a hard
> time to look for similar issues in other packages.

Code generation bugs are hard to detect at runtime. There are absolutely things you can do as a developer in your project to try to detect anomalies at run time like this, but not every project has written things so defensively. So I think it's honestly hard to actually make this "fail better" in this situation without projects making defensive changes. 

> 
> 1. llvm is very slow to compile, so when compiling rustc / cargo is good to
> use the llvm from the system to gain some speed.

Correct - but we can speed it up with the use of sccache in OBS, which I have been trying to have included in our OBS instances (if you know any OBS admins/maintainers, please get them to contact me because I can't seem to get in touch with any of them ...)

> 
> 2. rustc is expected to work with some range of llvm versions

"yes-ish". Even if it works, we still need to focus on "what received the most testing". Software always is reflective of, and works best in the way it was developed, and for rust that means lock-stepping llvm to rust versions, because that's how it's developed.

From now, I intend to pair rust versions to llvm versions rather than allowing it to change on us mid-release of a rust cycle like happened here, which introduced this chaos.

The reason we have to build llvm in leap15.3 is just because llvm13 isn't available there yet, else we could use the distro llvm. I can't remember the exact details about llvm and out-of-band updates in 15.3 but if 13 becomes available we can swap to that potentially. 

> I think that they are fair points, and TW can help in detecting early issues.
> 
> But still I think that there should be an outcome of this bug, like working
> on enabling the tests inside the %check section on the rust package.

That's an option, but it's also another time consuming option. OBS already is excruciatingly slow to build rust taking 2hrs or more - and that's with distro llvm, not the bundled llvm. Enabling the tests will likely only make this worst. (Contrast my home lab, where I can build rust in 40 mins with cold sccache and 10 minutes with hot cache). 

we could do the checks just on package submit time instead as an option, but then we'd only be testing on x86_64 (for now, I intend to get aarch64 hardware soon). 

Another option to consider too is extended openqa testing of firefox *as well* as the rust checks. Clearly this was missed and every time we see a failure like this, we can potential grow openqa to detect issues like this. 

Anyway, the rust1.55 updates are accepted and are in the Factory/Tumbleweed pipeline at the moment, and once that's accepted, I have staged the 1.55 updates for 15.3 in SLE, which will then flow out to leap. So I think this bug should probably switch focus to proactive steps to detect and prevent issues like this.

(In reply to William Brown from comment #18)
> Code generation bugs are hard to detect at runtime. There are absolutely
> things you can do as a developer in your project to try to detect anomalies
> at run time like this, but not every project has written things so
> defensively. So I think it's honestly hard to actually make this "fail
> better" in this situation without projects making defensive changes. 
Agreed, miscompilations are pretty ugly. If you add defensive coding, then possibly the issue disappears for unrelated reasons because you're not triggering the bug anymore. Had to debug such an issue once where the miscompilation disappeared with enabled assertions.

The typical cause of miscompilations is some inconsistency in LLVM's IR model or its usage in clients. Now that model isn't precisely specified and changes between versions. (Though of course with some care.)

Some inconsistencies go a bit deeper and are hard to fix [1]: "The LLVM community has dealt with these issues by usually fixing a problem when it can be demonstrated to lead to an *end-to-end miscompilation*: an optimizer malfunction that leads to the wrong machine code being emitted for some legal source program. The underlying, long-standing problems—and, indeed, a few opportunities
for end-to-end miscompilations—remain unsolved."

Of course we'd like to have a formally proven model, but I guess things change too fast for that, and of course many ideas haven't been fully fleshed out yet. Though at least some formal tools for peephole optimizations are used in LLVM development. [2]

> > 1. llvm is very slow to compile, so when compiling rustc / cargo is good to
> > use the llvm from the system to gain some speed.
> 
> Correct - but we can speed it up with the use of sccache in OBS, [...]
It's not just that, to be fair, but also about unbundling.

> The reason we have to build llvm in leap15.3 is just because llvm13 isn't
> available there yet, else we could use the distro llvm. I can't remember the
> exact details about llvm and out-of-band updates in 15.3 but if 13 becomes
> available we can swap to that potentially. 
That's not up to me—someone at SUSE would have to do that. So far we've only added new LLVMs with new Leap releases, but I don't see why it has to be that way. We could certainly add newer versions between releases as well.

Though it could lead to issues with the installation medium, perhaps that's why we want to limit the versions in Leap. In Tumbleweed we mostly switch everything to newer versions when they become available, but on SLE packages aren't rebuild when dependencies update, so we might need multiple libLLVMs on the DVD.

> we could do the checks just on package submit time instead as an option, but
> then we'd only be testing on x86_64 (for now, I intend to get aarch64
> hardware soon). 
In Python packages there is often some kind of multibuild setup that runs the tests separately from the actual package. Perhaps that could work?

[1] https://www.cs.utah.edu/~regehr/papers/undef-pldi17.pdf
[2] https://github.com/AliveToolkit/alive2

> > > 1. llvm is very slow to compile, so when compiling rustc / cargo is good to
> > > use the llvm from the system to gain some speed.
> > 
> > Correct - but we can speed it up with the use of sccache in OBS, [...]
> It's not just that, to be fair, but also about unbundling.

I'm aware :) 

> 
> > The reason we have to build llvm in leap15.3 is just because llvm13 isn't
> > available there yet, else we could use the distro llvm. I can't remember the
> > exact details about llvm and out-of-band updates in 15.3 but if 13 becomes
> > available we can swap to that potentially. 
> That's not up to me—someone at SUSE would have to do that. So far we've only
> added new LLVMs with new Leap releases, but I don't see why it has to be
> that way. We could certainly add newer versions between releases as well.

IIRC it's related to LLVM being part of the graphics pipeline somewhere but I don't recall all the details ... 

> 
> Though it could lead to issues with the installation medium, perhaps that's
> why we want to limit the versions in Leap. In Tumbleweed we mostly switch
> everything to newer versions when they become available, but on SLE packages
> aren't rebuild when dependencies update, so we might need multiple libLLVMs
> on the DVD.

Again, I think here it's up to what SLE does given how llvm/rust are managed there. I have a hand in Rust inside SUSE, but I'm not involved in the LLVM side. Regardless, I think keeping llvm lock stepped to rust versions will be important from now, and the question is how we go about that with SLE with regard to bundling LLVM or not. But that's something for me to chase up :) 

> 
> > we could do the checks just on package submit time instead as an option, but
> > then we'd only be testing on x86_64 (for now, I intend to get aarch64
> > hardware soon). 
> In Python packages there is often some kind of multibuild setup that runs
> the tests separately from the actual package. Perhaps that could work?

There is also a way to specify in OBS to run checks vs not so I think that's an option I need to look into.

(In reply to William Brown from comment #20)
> IIRC it's related to LLVM being part of the graphics pipeline somewhere but
> I don't recall all the details ... 
Usually Mesa wants a newer LLVM and that's how it ends up in SLE. But I'd ship the newest version at the time of release in Leap even if it wasn't for Mesa.

> > Though it could lead to issues with the installation medium, perhaps that's
> > why we want to limit the versions in Leap. In Tumbleweed we mostly switch
> > everything to newer versions when they become available, but on SLE packages
> > aren't rebuild when dependencies update, so we might need multiple libLLVMs
> > on the DVD.
> 
> Again, I think here it's up to what SLE does given how llvm/rust are managed
> there. I have a hand in Rust inside SUSE, but I'm not involved in the LLVM
> side. Regardless, I think keeping llvm lock stepped to rust versions will be
> important from now, and the question is how we go about that with SLE with
> regard to bundling LLVM or not. But that's something for me to chase up :) 
I don't have any concerns with that, given that Rust is usually quick to catch up. On the contrary, it makes it easier for me to bring newer LLVMs into TW.

As for SLE/Leap, let's see how we handle that.

I see two possible reasons why things didn't work out this time:

* Normally Rust doesn't compile with newer LLVMs because of interface changes, but this time that hasn't been the case. The relevant PR https://github.com/rust-lang/rust/pull/87570 was not in 1.55, yet we didn't have any compiler errors. With previous LLVM updates I saw that Rust didn't build and then I either ported back the "Upgrade to LLVM XX" PR or we waited for the next release to come out. This time we didn't see any errors, so we didn't do anything.

* The new pass manager is now the "default". Rust switched in https://github.com/rust-lang/rust/pull/88243, also in 1.56. The old PM is still maintained, but probably not as well-tested anymore since Clang made the switch.

> Usually Mesa wants a newer LLVM and that's how it ends up in SLE. But I'd
> ship the newest version at the time of release in Leap even if it wasn't for
> Mesa.

Yep, thought it was something like this. But again, I'm not really involved in that side of things .... 

> 
> I see two possible reasons why things didn't work out this time:
> 
> * Normally Rust doesn't compile with newer LLVMs because of interface
> changes, but this time that hasn't been the case. The relevant PR
> https://github.com/rust-lang/rust/pull/87570 was not in 1.55, yet we didn't
> have any compiler errors. With previous LLVM updates I saw that Rust didn't
> build and then I either ported back the "Upgrade to LLVM XX" PR or we waited
> for the next release to come out. This time we didn't see any errors, so we
> didn't do anything.
> 
> * The new pass manager is now the "default". Rust switched in
> https://github.com/rust-lang/rust/pull/88243, also in 1.56. The old PM is
> still maintained, but probably not as well-tested anymore since Clang made
> the switch.


I think 87570 is the likely missing piece, but it's a good lesson that we don't just need a minimum llvm version for rust, but also a maximum one because anything newer may not be tested or supported. So again, the lock-step will resolve that and reduce our potential surface area to issues.

As an FYI, I have now run a testbuild of firefox94 using the new rust1.56 compiler which is using llvm13 from the distro and ran the same tests I did before. As a result, firefox94 works as it should, no strange hangs or whatsoever anymore!

During these tests I found a strange thing in OBS, though. "osc buildinfo home:manfred-h:mozilla firefox94 openSUSE_Tumbleweed x86_64 | egrep -i 'cargo|rust|llvm|gcc'" shows that "osc build ..." would still pick up rust1.55 from openSUSE_Tumbleweed, even though rust1.56 could be found at https://download.opensuse.org/repositories/home:/manfred-h:/devel:/languages:/rust:/rus-1.56/openSUSE_Tumbleweed/
That's why I did the testbuild with the usual "build" script with a local copy of snapshot 20211102.

(In reply to Manfred Hollstein from comment #23)
> During these tests I found a strange thing in OBS, though. "osc buildinfo
> home:manfred-h:mozilla firefox94 openSUSE_Tumbleweed x86_64 | egrep -i
> 'cargo|rust|llvm|gcc'" shows that "osc build ..." would still pick up
> rust1.55 from openSUSE_Tumbleweed, even though rust1.56 could be found at
> https://download.opensuse.org/repositories/home:/manfred-h:/devel:/languages:
> /rust:/rus-1.56/openSUSE_Tumbleweed/
> That's why I did the testbuild with the usual "build" script with a local
> copy of snapshot 20211102.

Found the reason: explicitly excluding the unwanted versions helped; I now have the following in my prjconf:

  Prefer: cargo1.56 -cargo1.55 -cargo1.54 -cargo1.53
  Prefer: rust1.56 -rust1.55 -rust1.54 -rust1.53

Before I only had:

  Prefer: cargo1.56
  Prefer: rust1.56

I see the llvm12 based rust1.55 has been released in TW 20211104, thank you all who investigated and found the workaround

rust1.55-1.55.0-3.1
Build Date  : Thu 04 Nov 2021 14:50:40 CDT

> rpm -q --changelog rust1.55 | head
* Thu Oct 28 2021 Manfred Hollstein <manfred.h@gmx.net>
- Add BuildRequire: llvm12-devel on Tumbleweed; see boo#1192067
- Change llvm_bundling to be required on x86_64, aarch64 and ppc64le
  on 15.3, too. Otherwise cargo segfaults when building
  MozillaFirefox-93.0

But we need to manually trigger a rebuild of Factory Firefox to fully fix the issue, as the version in this snapshot is still old:

MozillaFirefox-93.0-3.2
Build Date  : Wed 03 Nov 2021 22:53:59 CDT
(in the build log:)
[   23s] [419/448] cumulate rust1.55-1.55.0-2.1

Can one of the factory maintainers please do this?

Manfred,

On Arch Linux, compiling Firefox 94 with Clang 13 + Rust 1.56.1 (LLVM 13) instead of Clang 12 + Rust 1.56.0 (LLVM 12) also reproduces this bug.

So it seems that this issue isn't restricted to Rust 1.55, and I don't understand your results with Rust 1.56.

(In reply to Jan Alexander Steffens from comment #27)
> On Arch Linux, compiling Firefox 94 with Clang 13 + Rust 1.56.1 (LLVM 13)
> instead of Clang 12 + Rust 1.56.0 (LLVM 12) also reproduces this bug.
In that case it would be good to know what exactly that miscompilation is. Whether it's in Rust or LLVM, upstream should probably be informed about this. (If they aren't already, but a quick search in bugs.llvm.org didn't yield anything.)

And if it's reproducible with Rust 1.56, why is Mozilla upstream not seeing it?

(In reply to Aaron Puchert from comment #28)
> (In reply to Jan Alexander Steffens from comment #27)
> > On Arch Linux, compiling Firefox 94 with Clang 13 + Rust 1.56.1 (LLVM 13)
> > instead of Clang 12 + Rust 1.56.0 (LLVM 12) also reproduces this bug.
> In that case it would be good to know what exactly that miscompilation is.
> Whether it's in Rust or LLVM, upstream should probably be informed about
> this. (If they aren't already, but a quick search in bugs.llvm.org didn't
> yield anything.)
> 
> And if it's reproducible with Rust 1.56, why is Mozilla upstream not seeing
> it?

I wonder if the difference is likely platform llvm 13 vs bundled llvm13 in rust. Most rust compilers from rustup would be bundled, not the system linked version, and this is nearly universally the preferred way for most developers and build pipelines to install rust today (see opensuse rust survey results).

Backporting https://hg.mozilla.org/mozilla-central/rev/85ab72b52e71 fixes this for me on Arch.

(In reply to Jan Alexander Steffens from comment #27)
> Manfred,
> 
> On Arch Linux, compiling Firefox 94 with Clang 13 + Rust 1.56.1 (LLVM 13)
> instead of Clang 12 + Rust 1.56.0 (LLVM 12) also reproduces this bug.
> 
> So it seems that this issue isn't restricted to Rust 1.55, and I don't
> understand your results with Rust 1.56.

You are right, I was probably too fast. The version I built locally here appeared to work OK, but the one built on OBS then behaved differently - == hung again. It just took quite some time until the package got built...

(In reply to Jan Alexander Steffens from comment #30)
> Backporting https://hg.mozilla.org/mozilla-central/rev/85ab72b52e71 fixes this for me on Arch.

Thanks, looks promising. Will try that out, too.

(In reply to Manfred Hollstein from comment #31)
> (In reply to Jan Alexander Steffens from comment #30)
> > Backporting https://hg.mozilla.org/mozilla-central/rev/85ab72b52e71 fixes this for me on Arch.
> 
> Thanks, looks promising. Will try that out, too.

Here's the backported patch, if it makes it easier: https://github.com/archlinux/svntogit-packages/blob/packages/firefox/trunk/0002-Bug-1735905-Upgrade-cubeb-pulse-to-fix-a-race-condit.patch

(In reply to Jan Alexander Steffens from comment #32)
> Here's the backported patch, if it makes it easier:
> https://github.com/archlinux/svntogit-packages/blob/packages/firefox/trunk/
> 0002-Bug-1735905-Upgrade-cubeb-pulse-to-fix-a-race-condit.patch

Thx, but I rebased the patch on my own in the meantime and built it using "osc build ...". The resulting binaries work correctly afaict. Build in OBS is currently running (just for openSUSE_Tumbleweed/x86_64). Here is my patch:

https://build.opensuse.org/package/view_file/home:manfred-h:mozilla/firefox94/mozilla-bmo1735905.patch?expand=1

Another FYI: the build of firefox94 using rust1.56 and llvm13 succeeded in

  <https://build.opensuse.org/package/show/home:manfred-h:mozilla/firefox94>

and the produced packages work flawlessly for me. In case anybody is going to test the packages, your feedback is highly appreciated!

(In reply to William Brown from comment #29)
> I wonder if the difference is likely platform llvm 13 vs bundled llvm13 in
> rust.
There isn't much of a meaningful difference, as I wrote in comment 1. If we need one of those patches, feel free to ping me or open a submit request. But a cursory reading suggests none of those are fixing miscompilations on x86.

> Most rust compilers from rustup would be bundled, not the system linked
> version, and this is nearly universally the preferred way for most developers
> and build pipelines to install rust today (see opensuse rust survey results).
Binaries from Mozilla are likely built with a bundled rust, but I think most Linux distros build their own Firefox with their own Rust, which in turn is built with their own LLVM.

(In reply to Jan Alexander Steffens from comment #30)
> Backporting https://hg.mozilla.org/mozilla-central/rev/85ab72b52e71 fixes
> this for me on Arch.
So it's not a miscompilation but a genuine bug that just didn't materialize with older LLVM versions? At least what we're seeing directly in the change is a regular variable being made atomic with load-release/store-acquire. Those atomic operations produce the same instruction sequence in x86 as their non-atomic counterparts, but the non-atomic operations can be more liberally reordered. Maybe that's what LLVM 13 was doing.

If we're sure that's the case I'd appreciate if someone changes the bug title accordingly. (I didn't test it yet.)

The Firefox 94 in Tumbleweed 20211110 is working perfect so far for the last 3 hours, including WebRTC (Jitsi) which used to hang right away earlier.

openSUSE-RU-2021:3936-1: An update that has one recommended fix and contains one feature can now be installed.

Category: recommended (moderate)
Bug References: 1192067
CVE References: 
JIRA References: SLE-19210
Sources used:
openSUSE Leap 15.3 (src):    rust-1.55.0-21.9.1, rust1.55-1.55.0-7.3.1

SUSE-RU-2021:3936-1: An update that has one recommended fix and contains one feature can now be installed.

Category: recommended (moderate)
Bug References: 1192067
CVE References: 
JIRA References: SLE-19210
Sources used:
SUSE Linux Enterprise Module for Development Tools 15-SP3 (src):    rust-1.55.0-21.9.1, rust1.55-1.55.0-7.3.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.