Bug 988273

Summary: Steam segmentation fault in libcrypto
Product: [openSUSE] openSUSE Tumbleweed Reporter: Dmitry Misharov <quarckster>
Component: X.OrgAssignee: Vítězslav Čížek <vcizek>
Status: RESOLVED FIXED QA Contact: E-mail List <xorg-maintainer-bugs>
Severity: Normal    
Priority: P2 - High CC: astieger, cyberalex4life, email, infopipe, jon, kolAflash, lnussel, maintenance, matthias, mpluskal, opensuse, osukup, pdvrosado, pedro.rosado, rombert, stephan.barth, vcizek, wagner-thomas
Version: Current   
Target Milestone: ---   
Hardware: All   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Dmitry Misharov 2016-07-09 07:41:08 UTC
User-Agent:       Mozilla/5.0 (X11; Linux x86_64; rv:47.0) Gecko/20100101 Firefox/47.0
Build Identifier: 

Steam crashes immediately after starting due this bug https://github.com/ValveSoftware/steam-for-linux/issues/4504

Reproducible: Always

Steps to Reproduce:
1. Install steam 1.0.0.52
2. Run it
Actual Results:  
Steam crashes with segmentation fault.


Expected Results:  
Steam works fine.

There is fix/workaround for this http://www.gearsongallium.com/?p=3276#utm_source=feedly&utm_medium=rss&utm_campaign=fix-for-segmentation-fault-in-libcrypto-steam-beta mesa must be compiled with libnettle:
--with-sha1=libnettle
Comment 1 Egbert Eich 2016-07-09 10:10:56 UTC
From looking at the discussion on Github, I don't understand why building Mesa with libnettle would help when steam running with the system provided libcryto - ie without the steam provided runtime librypto - doesn't work.

This seems to be a libcrypto issue: Before we do any changes in the built of Mesa, the  libcrypto & steam maintainers should have a look at this.
Reassigning to the maintainer of openssl.
Comment 2 Ondřej Súkup 2016-07-09 10:38:11 UTC
@Egbert, SR#407423 for TW and SR#407424 for Leap:42.1 is workaround with libnettle + update to 12.0.1 / 11.0.9

comment by Marek Olsak is clear : BTW, this is libcrypto developers' fault, because they had changed their public interface, which is the cause of this incompatibility.

and because steam isn't in our control ..
Comment 3 Bernhard Wiedemann 2016-07-09 12:00:16 UTC
This is an autogenerated message for OBS integration:
This bug (988273) was mentioned in
https://build.opensuse.org/request/show/407424 42.1 / Mesa
Comment 4 Egbert Eich 2016-07-09 14:43:34 UTC
(In reply to Ondřej Súkup from comment #2)
> @Egbert, SR#407423 for TW and SR#407424 for Leap:42.1 is workaround with
> libnettle + update to 12.0.1 / 11.0.9
> 
> comment by Marek Olsak is clear : BTW, this is libcrypto developers' fault,
> because they had changed their public interface, which is the cause of this
> incompatibility.

WHich incompatibility?
When the run time libs shipped by steam are not used  /lib/libcrypto.so.1.0.0 is used by both Mesa and steam. Where do you see an ABI breakage?

I would like the maintainer of openssl (where libcryto comes from) to look into this issue before we change anything in Mesa.

> and because steam isn't in our control ..

Of course not. But is this a valid reason to do kludge around this by adapting other packages?

I've declined the SRs for Mesa for now due to the change of crypto libs for now.
I'd like to understand the issue better.
I'm fine with the other updates.
Comment 5 Ondřej Súkup 2016-07-10 19:12:04 UTC
(In reply to Egbert Eich from comment #4)
> (In reply to Ondřej Súkup from comment #2)
> > @Egbert, SR#407423 for TW and SR#407424 for Leap:42.1 is workaround with
> > libnettle + update to 12.0.1 / 11.0.9
> > 
> > comment by Marek Olsak is clear : BTW, this is libcrypto developers' fault,
> > because they had changed their public interface, which is the cause of this
> > incompatibility.
> 
> WHich incompatibility?
> When the run time libs shipped by steam are not used 
> /lib/libcrypto.so.1.0.0 is used by both Mesa and steam. Where do you see an
> ABI breakage?

i think steam binary uses steamclient.so and steamservice.so with integrated libcrypto into this two libraries ... and here might be incompactibility between steam and Mesa used libs

> I would like the maintainer of openssl (where libcryto comes from) to look
> into this issue before we change anything in Mesa.
> 
> > and because steam isn't in our control ..
> 
> Of course not. But is this a valid reason to do kludge around this by
> adapting other packages?
> 
> I've declined the SRs for Mesa for now due to the change of crypto libs for
> now.
> I'd like to understand the issue better.
> I'm fine with the other updates.
Comment 6 Egbert Eich 2016-07-11 06:45:00 UTC
(In reply to Ondřej Súkup from comment #5)
> 
> i think steam binary uses steamclient.so and steamservice.so with integrated
> libcrypto into this two libraries ... and here might be incompactibility
> between steam and Mesa used libs
> 

What do you mean? Statically linked? I very much doubt this.

It could well be that we are seeing different versions of the same lib loaded and functions from different versions being called for the same sha1 calculation.
What I still don't understand is that this also happens when nut using the steam provided runtime.
Unless the openssl maintainer is aware of any issue with a recent version of libcrypto, this needs to be investigated more thoroughly - by the steam package maintainers: presently, I don't have the time for this.
Comment 7 Vítězslav Čížek 2016-07-11 15:48:05 UTC
(In reply to Egbert Eich from comment #6)
> What do you mean? Statically linked? I very much doubt this.
They statically link openssl in ~/.local/share/Steam/ubuntu12_32/steamclient.so
(https://github.com/ValveSoftware/steam-for-linux/issues/3803)
Comment 8 Andreas Stieger 2016-07-17 20:01:11 UTC
Not seeing an action for maintenance just now (request review declined by maintainer). Removing needinfo flag.
Comment 9 Dmitry Misharov 2016-07-23 10:11:53 UTC
Almost two weeks have passed and still there is no progress. Steam doesn't work all that time, though the solution was provided. What can I do to speedup issue resolving?
Comment 10 Matthias Mailänder 2016-07-23 10:24:44 UTC
Does replacing libcrypto with libnettle using the patched Mesa package from https://build.opensuse.org/package/show/home:boombatower:tumbleweed++/Mesa resolve it for you?
Comment 11 Dmitry Misharov 2016-07-23 10:35:44 UTC
It does, but I think this fix should be presented in default repo.
Comment 12 Egbert Eich 2016-07-23 11:18:54 UTC
(In reply to Dmitry Misharov from comment #11)
> It does, but I think this fix should be presented in default repo.

No.

The claimed 'solution' wasn't a solution but a workaround. It was not even clear why the workaround actually worked as this was never investigated.
In comment #5 and #7 it is claimed that steam statically links against libcrypto. The explanation given in the link provided in comment #7 explains the situation around statically linked libs and provides a workaround which doesn't require to touch other components of the system.
Yet, the backtraces shown in the links provided in the original bug description contradict this static linking. 
Again, we cannot fix this issue in Mesa - we will not touch system components to work around issues with proprietary 3rd party applications. It is just wrong and  *will* *not* *happen*.
Some other workaround needs to be found which will remain local to steam. Someone just needs to investigate this thoroughly and come up with a full explanation and sensible workaround. I don't have the time for this presently - sorry.
Comment 13 Matthias Mailänder 2016-07-23 12:18:52 UTC
I agree in principle that workarounds should be kept localized, but is there a reason why compiling Mesa with libnettle is a bad thing? The ABI breakage in libcrypto might cause other closed source applications to crash.
Comment 14 Ondřej Súkup 2016-07-27 06:16:30 UTC
(In reply to Matthias Mailänder from comment #13)
> I agree in principle that workarounds should be kept localized, but is there
> a reason why compiling Mesa with libnettle is a bad thing? The ABI breakage
> in libcrypto might cause other closed source applications to crash.

because it overrides fixes for openSSL and creates bigger problem with maintenance.. + bug is on Steam/Valve side
Comment 15 Marcus Meissner 2016-07-28 09:50:13 UTC
the backtrace shows it embdes a copy of libcrypto.so.1.0.0 ...

canb this be removed perhaps and the system one used, perhaps this helps already
Comment 16 Matthias Mailänder 2016-07-28 19:56:14 UTC
Neither

find ~/.local/share/Steam/ubuntu12_32/steam-runtime -type f -name "libcrypto*" -print -delete

nor

find ~/.local/share/Steam/ubuntu12_32/steam-runtime -type f -name "libgpg*" -print -delete

as suggested in https://bugs.archlinux.org/task/48994 did help on my TW system.
Comment 17 Pedro Rosado 2016-09-03 16:38:43 UTC
*** Bug 996928 has been marked as a duplicate of this bug. ***
Comment 18 Pedro Rosado 2016-09-03 16:51:37 UTC
I've marked my bug as a duplicate of this since it's the same, segmentation fault at line 713. This affects both TW and Leap 42.2 beta. Many people suffer from this, and Steam is still the main software for those who play on linux, not having a fix for this is a shame for all who use Opensuse. I've contacted Valve, and on their side, their response is to use ubuntu 12.04 or any other Buntu OS. So a fix needs to come from the community...
Comment 19 Matthias Mailänder 2016-09-04 07:10:53 UTC
It should definitely be added to the 42.2 Release Notes so people who want to use Steam for gaming know that they can't upgrade yet.
Comment 20 Matthias Mailänder 2016-09-04 09:29:08 UTC
On Reddit it was suggested that we look into the command line version as a temporary alternative and for server admins. I started experimenting with it at https://build.opensuse.org/package/show/home:Mailaender:branches:games:tools/steamcmd
Comment 21 Jon Brightwell 2016-09-04 10:12:53 UTC
I'm not 100% sure on how the conflict causes the seg fault with static vs dynamic so excuse any ignorance.

openssl 1.1.0 is out now. Has anyone had the chance to check it is still an issue? What specifically changed with libcrypto to break it, I couldn't see anything in the changelog?

Why is libnettle such a bad idea? It's already in the official repos and the only use of libnettle in mesa is for SHA1 which is AFAIK only used by shader code. Not exactly security centric and I couldn't find any openssl specific patches on mesa's obs.

There's about 4 compiles of mesa with libnettle in obs and I've been using boomba's for several months now without any adverse affects.
Comment 22 Marcus Meissner 2016-09-04 21:21:03 UTC
- tumbleweed ... i fixed up the cpuid call when using the system libraries, but then it also crashes in RSA_free -> ENGINE_finish with a corrupted pointer
Comment 23 Marcus Meissner 2016-09-06 09:19:09 UTC
I debugged this more.

The secondary crash (besides the cpuid one).

The binary "steam" has a static libcrypto.a built in. RSA_new() is called in it.
This initializes all of the RSA struct, except the "engine" pointer, because engines are not built in the static build.

While RSA_new is called in this static build, RSA_free is called in the shared library, which has engine support. There it tries to dereference RSA->engine if it is not NULL and that crashes.


I went an opened a steam issue.

I also went and opened pull requests for openssl to initialize RSA->engine to NULL, for 1.0.1 and 1.0.2 branch, so if that is accepted and published and steam picks it up, it will in their static builds too.
Comment 25 Dmitry Misharov 2016-09-25 16:47:56 UTC
The latest snapshot has fixed the issue. Thanks to everyone who helped with the resolving. You are awesome.
Comment 26 Matthias Mailänder 2016-09-26 05:06:40 UTC
Indeed, great work! To solve this once and for all we may need to build every library from the Steam runtime from source https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=steam-libs as relying on the Ubuntu 12.04 ones is a fundamentally broken concept from Valve. I added a warning to Tumbleweed users to https://en.opensuse.org/Steam for now.
Comment 27 Matthias Mailänder 2016-10-19 19:22:52 UTC
https://www.reddit.com/r/openSUSE/comments/5867hg/steam_cs_go_working_again_in_tumbleweed/d8ycve8 Some compliments for the hero of the openSUSE gaming community. :)
Comment 28 kolA flash 2016-11-19 08:50:42 UTC
For me this seems to be still broken on openSUSE 42.2.
Anyone else tested this on 42.2?

So either please:
- please reopen this bug until it's fixed for 42.2
OR
- bug 996928 isn't a duplicate, because it was specifically for 42.2
Comment 29 Dmitry Misharov 2016-11-19 09:07:47 UTC
Steam works on openSUSE 42.2. So for now the status of this issue should be stay as is.
Comment 30 Alex Petrini 2017-01-09 13:52:02 UTC
Take a look at https://bugzilla.opensuse.org/show_bug.cgi?id=1017891, I wasn't able to fix steam for Star Conflict or Strife. This bug is still present in 42.2.
Steam works for Valve: Robocraft, Team Fortress and probably others, for War Thunder. This bug is not fixed yet. Please investigate!
Comment 31 kolA flash 2017-05-03 10:28:59 UTC
(In reply to kolA flash from comment #28)
> For me this seems to be still broken on openSUSE 42.2.
> Anyone else tested this on 42.2?
> 
> So either please:
> - please reopen this bug until it's fixed for 42.2
> OR
> - bug 996928 isn't a duplicate, because it was specifically for 42.2

Sorry for my previous comment.
I didn't installed Steam from the openSUSE-42.2-NonOSS repo, so my Steam didn't used the Steam fixing stuff.

https://build.opensuse.org/package/view_file/openSUSE:Leap:42.2:NonFree:Update/steam.6312/steamruntime-fix?expand=1
(line 43 - 60)

http://download.opensuse.org/distribution/leap/42.2/repo/oss/suse/x86_64/libopenssl1_0_0-steam-1.0.2h-1.2.x86_64.rpm

http://download.opensuse.org/distribution/leap/42.2/repo/oss/suse/x86_64/libopenssl1_0_0-steam-32bit-1.0.2h-1.2.x86_64.rpm

I manually linked those libraries into a non-YaST Steam installation and now that Steam is working fine on 42.2!
Comment 32 Swamp Workflow Management 2018-03-14 07:30:33 UTC
This is an autogenerated message for OBS integration:
This bug (988273) was mentioned in
https://build.opensuse.org/request/show/586663 15.0:NonFree / steam