Bug 1194660 - Leap 15.3 ISO: Installation does not find repositories
Leap 15.3 ISO: Installation does not find repositories
Status: IN_PROGRESS
Classification: openSUSE
Product: openSUSE Distribution
Classification: openSUSE
Component: Installation
Leap 15.3
aarch64 Other
: P2 - High : Major (vote)
: ---
Assigned To: Lubos Kocman
Jiri Srain
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2022-01-13 18:08 UTC by Axel Braun
Modified: 2022-02-15 16:01 UTC (History)
7 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
y2logs (5.43 MB, application/x-compressed-tar)
2022-01-14 10:13 UTC, Axel Braun
Details
where it hangs... (5.08 MB, image/jpeg)
2022-01-14 16:35 UTC, Axel Braun
Details
during installation (7.23 MB, image/jpeg)
2022-01-14 18:04 UTC, Axel Braun
Details
curl -vvv (7.09 MB, image/jpeg)
2022-01-18 10:02 UTC, Axel Braun
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Axel Braun 2022-01-13 18:08:39 UTC
In current Leap 15.3 build for aarch64
- when booting from NET-ISO, system hangs at Adding Repositories -> Download documentation
- when booting from DVD, when reading list of online repositories (Unable to download list of repositories or no repositories defined)

(Yes, there is an online connection ;-)
Comment 1 Andreas Stieger 2022-01-13 20:40:34 UTC
Does this hang on a particular one? Like in bug 1194626?
Comment 2 Axel Braun 2022-01-14 07:28:28 UTC
(In reply to Andreas Stieger from comment #1)
> Does this hang on a particular one? Like in bug 1194626?

No, it does not find any online repository. Impact of the new naming scheme maybe?
Comment 3 David Diaz 2022-01-14 08:37:53 UTC
Axel, could you please attach YaST logs?

You can collect them by running save_y2logs command. See https://en.opensuse.org/openSUSE:Report_a_YaST_bug for more information.

Thanks in advance.
Comment 4 Axel Braun 2022-01-14 10:13:55 UTC
Created attachment 855285 [details]
y2logs

this is the y2log of the DVD installation (for obvious reasons, the one from NET Image cant be provided...)
Comment 5 David Diaz 2022-01-14 15:03:18 UTC
Could it be a temporary problem? I have tried using openSUSE-Leap-15.3-2-NET-aarch64-Build24.5-Media in a virtual machine and the installation worked with no other issue than the slowness of the emulation.

Do you mind retry it?
Comment 6 Axel Braun 2022-01-14 16:35:57 UTC
Created attachment 855300 [details]
where it hangs...

Hello David,
I tried with 2 different Raspi3, connected to ethernet, and both installations got stuck at the same point, see attached.
Network connection has been verified as working beforehand.
Can you submit the URL that this step is calling?
Comment 7 David Diaz 2022-01-14 16:43:32 UTC
(In reply to Axel Braun from comment #6)
> Created attachment 855300 [details]
> where it hangs...
> 
> Hello David,
> I tried with 2 different Raspi3, connected to ethernet, and both
> installations got stuck at the same point, see attached.
> Network connection has been verified as working beforehand.
> Can you submit the URL that this step is calling?

I'll check it.

BTW, from there you can collect the logs by running the save_y2logs either, switching to console 2 with Ctrl-Alt-F2 or from an X terminal window by pressing Ctrl-Alt-Shift-X. See https://en.opensuse.org/openSUSE:Report_a_YaST_bug#How_can_I_issue_shell_commands_during_an_installation.3F
Comment 8 David Diaz 2022-01-14 17:33:43 UTC
(In reply to Axel Braun from comment #6)
> Created attachment 855300 [details]
> where it hangs...
> 
> Hello David,
> I tried with 2 different Raspi3, connected to ethernet, and both
> installations got stuck at the same point, see attached.
> Network connection has been verified as working beforehand.
> Can you submit the URL that this step is calling?

https://download.opensuse.org/YaST/Repos/openSUSE_Leap_15.3_Servers.xml

See https://github.com/yast/skelcd-control-openSUSE/blob/546c28c7fbd060bceafa73208b51d994b8366b0b/control/control.openSUSE.xml#L101-L102
Comment 9 David Diaz 2022-01-14 17:43:25 UTC
From your attached logs, 

> 4234:2021-11-05 08:40:02 <1> install(3307) [Ruby] clients/inst_productsources.rb(ReadControlFile):446 Using link: https://download.opensuse.org/YaST/Repos/openSUSE_Leap_15.3_Servers.xml
> ...
> 4238:2021-11-05 08:40:02 <1> install(3307) [Ruby] clients/inst_productsources.rb(DownloadFile):492 Server response: nil
> 4239-2021-11-05 08:40:02 <3> install(3307) [Ruby] clients/inst_productsources.rb(DownloadAndParseSources):579 Unable to download list of online repositories

And that's the reason of your problem [1] 

So, I fear something is going wrong on your side when trying to download the file.

[1] https://github.com/yast/yast-packager/blob/bff0d8babd17eff9951554f4607707738298181e/src/lib/y2packager/clients/inst_productsources.rb#L572-L575
Comment 10 Axel Braun 2022-01-14 18:04:28 UTC
Created attachment 855307 [details]
during installation

looks like $releasever is not defined at that point in time (or am I doing something wrong here?
Comment 11 Axel Braun 2022-01-14 18:15:26 UTC
I get a certificate error when doing
wget https://download.opensuse.org/YaST/Repos/openSUSE_Leap_15.3_Servers.xml
from the window during installation, so wget breaks.
How does YaST the download?
Comment 12 Axel Braun 2022-01-16 08:07:29 UTC
Stefan, ist Not only tie NET ISO
Comment 13 David Diaz 2022-01-16 19:16:38 UTC
(In reply to Axel Braun from comment #10)
> Created attachment 855307 [details]
> during installation
> 
> looks like $releasever is not defined at that point in time (or am I doing
> something wrong here?

But your problem is in an already expanded (actually, hard-coded in the control file) URL. 

>  Using link: https://download.opensuse.org/YaST/Repos/openSUSE_Leap_15.3_Servers.xml

So, $releasever is not involved (or maybe it's me who is missing something)
Comment 14 David Diaz 2022-01-16 19:47:57 UTC
(In reply to Axel Braun from comment #11)
> I get a certificate error when doing
> wget https://download.opensuse.org/YaST/Repos/openSUSE_Leap_15.3_Servers.xml
> from the window during installation, so wget breaks.
> How does YaST the download?

YaST is using Yast::HTTP.Get[1], which is part of the yast-transfer agent and it is using curl/libcurl behind the scene.

And this was a great question to make me realize that I overlooked one line in your attached logs

> 4237  2021-11-05 08:40:02 <3> install(3307) [agent-curl] CurlAgent.cc(Get):129 curl told us 60

and man curl.1 tells

> 60     Peer certificate cannot be authenticated with known CA certificates.


@Ladislav, does it ring a bell to you?

[1] https://github.com/yast/yast-transfer/blob/a967d8bcce95ede785aeb6783c1dba6feeecc3aa/src/modules/HTTP.rb#L41-L51
[2] https://github.com/yast/yast-transfer/blob/a967d8bcce95ede785aeb6783c1dba6feeecc3aa/agent-curl/src/CurlAgent.cc#L87-L178
Comment 15 Ladislav Slezák 2022-01-17 16:47:00 UTC
Hm, it works fine for me on an x86_64...

You might try running this to get more details about the failure:

  curl -vvv https://download.opensuse.org/YaST/Repos/openSUSE_Leap_15.3_Servers.xml" 


Also check that the CA certificates package is present in the inst-sys:

  grep certificate /.packages*

It should list the "ca-certificates-mozilla" package.
Comment 16 Axel Braun 2022-01-18 10:02:57 UTC
Created attachment 855364 [details]
curl -vvv

Hello Ladislav, find the screenshot of the curl -vvv command
'certificate is not yet valid' indicates that it is too young.
date command issues Fri 05 Nov 2021 -> no sync with ntp so far?
I set the date manually to 2022-01-18, and the download was OK!
Comment 17 Guillaume GARDET 2022-01-18 11:07:58 UTC
So, all systems without RTC or with broken date and time are affected.
The installer could adjust time with an NTP call, maybe?
Comment 18 Lukas Ocilka 2022-01-18 11:23:01 UTC
(In reply to Guillaume GARDET from comment #17)
> So, all systems without RTC or with broken date and time are affected.
> The installer could adjust time with an NTP call, maybe?

That's a nice idea, but there is a problem:
- SLE can't use openSUSE NTP servers (or other public ones)
- Some installation can't access external networks
- Some must not at all

I believe we have some kind of a time-stamp in the inst-sys (date of build) that could be used for comparison with the system timedate (Steffen?). There in fact already is a dialog for NTP sync and setup.
Comment 19 Axel Braun 2022-01-18 11:31:27 UTC
(In reply to Lukas Ocilka from comment #18)
> (In reply to Guillaume GARDET from comment #17)
> > So, all systems without RTC or with broken date and time are affected.
> > The installer could adjust time with an NTP call, maybe?
> 
> That's a nice idea, but there is a problem:
> - SLE can't use openSUSE NTP servers (or other public ones)

Why not?

> - Some installation can't access external networks

Then they cant add online repositories, so it's a don't care

> - Some must not at all

Must not at all what?

> I believe we have some kind of a time-stamp in the inst-sys (date of build)
> that could be used for comparison with the system timedate (Steffen?). There
> in fact already is a dialog for NTP sync and setup.

Yes, but that comes much later in the installation process
Comment 20 Andreas Stieger 2022-01-18 11:42:56 UTC
(In reply to Axel Braun from comment #19)
> (In reply to Lukas Ocilka from comment #18)
> > - SLE can't use openSUSE NTP servers (or other public ones)
> 
> Why not?

https://www.ntppool.org/en/vendors.html
Comment 21 Steffen Winterfeldt 2022-01-18 11:55:07 UTC
There is aleady code that adjusts the time to some reasonable value:

https://github.com/openSUSE/installation-images/blob/master/data/root/etc/inst_setup#L23-L36

basically to the release date.

That doesn't help of course if the certificate has been renewed later.

Using NTP without at least having set the timezone before might make things
worse for most.

So an option would be to set via NTP if the date is off by at leat a day or so.
Seems a bit on the overengineering side, honestly.

OTOH, simply ignoring the date for the certificate check looks not really
more insecure to me.

BTW, linuxrc ignores the date when running gpg to check signatures for
exactly this reason.
Comment 22 Lukas Ocilka 2022-01-18 12:03:01 UTC
(In reply to Axel Braun from comment #19)
> > That's a nice idea, but there is a problem:
> > - SLE can't use openSUSE NTP servers (or other public ones)
> 
> Why not?
> 
> > - Some installation can't access external networks
> 
> Then they cant add online repositories, so it's a don't care

They can, by using SMT / RMT

> > - Some must not at all
> 
> Must not at all what?

Use external network - by a company policy or so. There are so many weird scenarios out there.

(In reply to Steffen Winterfeldt from comment #21)
> OTOH, simply ignoring the date for the certificate check looks not really
> more insecure to me.
> 
> BTW, linuxrc ignores the date when running gpg to check signatures for
> exactly this reason.

All in all, it's a new feature request. I'm not saying that because I would not like to have this implemented, but simply because there are too many unknowns and security team must be involved.

Maybe we could have this for nonRTC systems only?
Comment 23 Axel Braun 2022-01-27 09:39:35 UTC
Maybe we just trigger a rebuild for the installation media to have a workaround for the current situation?
Comment 24 Steffen Winterfeldt 2022-01-27 10:06:56 UTC
Using a rebuilt installation-images, yes, that would do.
Comment 25 Ladislav Slezák 2022-01-31 15:56:09 UTC
I do not know who can trigger installation-images rebuild for 15.3 aarch64, Lubos?
Comment 26 Stefan Hundhammer 2022-02-03 21:35:53 UTC
If this is the same machine with only 1 GB RAM for attempting an installation from a NET ISO, IMHO this seems to be the same problem as in bug #1194832:

Simply running out of RAM because the downloaded data can't be stored on the RAM disk anymore. See also bug #1194832 comment #8.
Comment 27 Axel Braun 2022-02-04 13:33:19 UTC
(In reply to Stefan Hundhammer from comment #26)
> If this is the same machine with only 1 GB RAM for attempting an
> installation from a NET ISO, IMHO this seems to be the same problem as in
> bug #1194832:
> 
> Simply running out of RAM because the downloaded data can't be stored on the
> RAM disk anymore. See also bug #1194832 comment #8.

No, the root cause of the issue is a different one, and applies as well for machines with plenty of RAM. See comment #17 below
Comment 28 Lubos Kocman 2022-02-15 16:00:51 UTC
Respin in progress https://build.opensuse.org/project/show/openSUSE:Leap:15.3:Update:Respin