Bug 1073141 - yast failed to download apparmor-docs-2.11.0-6.2 while repo has apparmor-docs-2.11.1-2.1
yast failed to download apparmor-docs-2.11.0-6.2 while repo has apparmor-docs...
Status: RESOLVED WORKSFORME
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: libzypp
Current
PowerPC Other
: P5 - None : Normal (vote)
: ---
Assigned To: E-mail List
E-mail List
http://9.101.18.90/tests/313/modules/...
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2017-12-15 17:51 UTC by Michel Normand
Modified: 2018-01-11 14:55 UTC (History)
2 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
install_and_reboot-y2logs.tar.bz2 (131.33 KB, application/x-bzip)
2017-12-15 17:51 UTC, Michel Normand
Details
yast_failure_missing_apparmor-doc.png (185.45 KB, image/png)
2017-12-15 17:57 UTC, Michel Normand
Details
repo_oss_apparmor_docs_2_11_1_2_1.png (341.61 KB, image/png)
2017-12-15 17:58 UTC, Michel Normand
Details
varcachezyppraw.tgz (4.52 MB, application/x-compressed-tar)
2017-12-19 18:12 UTC, Michel Normand
Details
varcachezyppraw.packages.tgz (8.35 MB, application/x-compressed-tar)
2017-12-19 18:14 UTC, Michel Normand
Details
yast2.tgz (137.26 KB, application/x-compressed-tar)
2017-12-19 18:16 UTC, Michel Normand
Details
start_install-y2logs-wo_zypp_cache-w_zypp_media_curl_debug_2.tgz (190.25 KB, application/x-compressed-tar)
2017-12-20 16:29 UTC, Michel Normand
Details
manual_curl_same_host.txt (1.58 KB, text/plain)
2017-12-20 16:56 UTC, Michel Normand
Details
manual_curl_same_host_tumbleweed.txt (1.44 KB, text/plain)
2017-12-20 18:42 UTC, Michel Normand
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Michel Normand 2017-12-15 17:51:33 UTC
Created attachment 753337 [details]
install_and_reboot-y2logs.tar.bz2

yast failed to download apparmor-docs-2.11.0-6.2 while repo has apparmor-docs-2.11.1-2.1

Trying a Net iso installation in openQA environment, but using default repo http://download.opensuse.org/ports/ppc/tumbleweed/repo/oss

There is obviously a discrepancy between the Net iso and the oss repo,
as reported  by attached files.
But I do not understand from y2log why Yast(or zypp) is looking for the  apparmor-docs-2.11.0-6.2.noarch.rpm

extract y2log
===
2017-12-15 10:18:07 <1> install(3217) [Ruby] modules/Slides.rb:88 Slideshow directory /tmp/YaST2-03217-ckBxcp/slidedir//txt/en does not exist
2017-12-15 10:18:07 <1> install(3217) [Ruby] modules/PackageSlideShow.rb:951 src #0: [4903040822]
2017-12-15 10:18:07 <1> install(3217) [Ruby] modules/PackageSlideShow.rb:1318 Package 'apparmor-docs' is remote
2017-12-15 10:18:07 <1> install(3217) [zypp++] DeltaCandidates.cc(deltaRpms):82 package: (169)apparmor-docs-2.11.0-6.2.noarch(openSUSE-20171214-0)
...
2017-12-15 10:42:43 <1> install(3217) [zypp++] MediaCurl.cc(doGetFileCopyFile):1499 URL: http://download.opensuse.org/ports/ppc/tumbleweed/repo/oss/suse/noarch/apparmor-docs-2.11.0-6.2.noarch.rpm
2017-12-15 10:42:43 <3> install(3217) [zypp] MediaCurl.cc(doGetFileCopyFile):1563 curl error: 22: The requested URL returned error: 404 Not Found, temp file size 0 bytes.
===


The openQA test failed with:
===
[Build 20171214] openQA test fails in install_and_reboot

## Observation

openQA test in scenario opensuse-Tumbleweed-NET-ppc64le-install_only@ppc64le fails in
[install_and_reboot](http://9.101.18.90/tests/313/modules/install_and_reboot/steps/2)
===
Comment 1 Michel Normand 2017-12-15 17:57:03 UTC
Created attachment 753339 [details]
yast_failure_missing_apparmor-doc.png
Comment 2 Michel Normand 2017-12-15 17:58:12 UTC
Created attachment 753340 [details]
repo_oss_apparmor_docs_2_11_1_2_1.png
Comment 3 Imobach Gonzalez Sosa 2017-12-19 12:00:57 UTC
Maybe there was a discrepancy between the packages index and the repository itself. Reassigning to the libzypp team.
Comment 4 Michael Andres 2017-12-19 13:46:13 UTC
(In reply to Imobach Gonzalez Sosa from comment #3)
> Maybe there was a discrepancy between the packages index and the repository
> itself. Reassigning to the libzypp team.

Yes, though this is not visible in the logs. Only the downloaded metadata could reveal this (i.e. the content of /var/cache/zypp/raw).

Either the server repo was updated shortly after yast read the (now old) metadata or one of the mirrors did still distribute the old metadata set.


In both cases you should not be able to reproduce this, unless we had broken mirror permanently delivering old data. In this case we'd need the content of /var/cache/zypp/raw from within the instsys to verify this.
Comment 5 Michel Normand 2017-12-19 18:12:51 UTC
Created attachment 753717 [details]
varcachezyppraw.tgz

/var/cache/zypp/raw without openSUSE-20171218-0/suse/setup/descr/packages.gz
(to reduce tarball size)
Comment 6 Michel Normand 2017-12-19 18:14:58 UTC
Created attachment 753718 [details]
varcachezyppraw.packages.tgz

only the /var/cache/zypp/raw/openSUSE-20171218-0/suse/setup/descr/packages.gz
(to reduce tarball size)
Comment 7 Michel Normand 2017-12-19 18:16:42 UTC
Created attachment 753719 [details]
yast2.tgz

the YaST2/ data associated to two above tarballs
Comment 8 Michel Normand 2017-12-20 06:56:59 UTC
(In reply to Michel Normand from comment #6)
> Created attachment 753718 [details]
> varcachezyppraw.packages.tgz
> 
> only the /var/cache/zypp/raw/openSUSE-20171218-0/suse/setup/descr/packages.gz
> (to reduce tarball size)

This packages.gz seems to be obsolete (KIWIROOT ... 20170924)
===
$head -n3 tmp/var/cache/zypp/raw/openSUSE-20171218-0/suse/setup/descr/packages
=Ver: 2.0
##----------------------------------------
## -d /usr/src/packages/KIWIROOT/main/openSUSE-20170924-ppc64-ppc64le-Build0001-Media1/suse -P -Z -C -K -M 3 -V -F -B -o /usr/src/packages/KIWIROOT/main/openSUSE-20170924-ppc64-ppc64le-Build0001-Media1/suse/setup/descr
===

So the question is: why yast2 is downloading this obsolete packages.gz that is not same as manually downloaded from http://download.opensuse.org/ports/ppc/tumbleweed/repo/oss/suse/setup/descr/packages.gz
Comment 9 Michael Andres 2017-12-20 10:58:49 UTC
YAST is downloading http://download.opensuse.org/ports/ppc/tumbleweed/repo/oss/suse/setup/descr/packages.gz, but the server (or proxy, or mirror) seems to deliver old data.

- Did you manually download  _on_the_machine_  that shows the error? Would be good to know which content `curl -v http://download.opensuse.org/.../packages.gz` delivers on this machine. (switching to the text console before canceling YAST)

- We're sure there's no fake download.opensuse.org entry in /etc/hosts which redirects to some dummy server? 

- There's no (transparent) proxy in use which could deliver old data?

Please set 'ZYPP_MEDIA_CURL_DEBUG=2' in the environment before starting YAST. The yast logs will then contain more information about he download.
Comment 10 Michel Normand 2017-12-20 16:29:42 UTC
Created attachment 753894 [details]
start_install-y2logs-wo_zypp_cache-w_zypp_media_curl_debug_2.tgz

As requested by comment#9 set 'ZYPP_MEDIA_CURL_DEBUG=2' before yast (as boot parameter passed to openQA)
The y2log in attached log is reporting a download from default server,
BUT the Last Modified date is not correct:
===
* Re-using existing connection! (#0) with host download.opensuse.org
* Connected to download.opensuse.org (195.135.221.134) port 80 (#0)
> GET /ports/ppc/tumbleweed/repo/oss/suse/setup/descr/packages.gz HTTP/1.1
> Host: download.opensuse.org
> User-Agent: ZYpp 17.0.5 (curl 7.57.0) openSUSE-Tumbleweed-ppc64le
> Accept: */*, application/metalink+xml, application/metalink4+xml
< HTTP/1.1 200 OK 
< Date: Wed, 13 Dec 2017 09:12:29 GMT
< Server: Apache/2.4.23 (Linux/SUSE)
< X-Prefix: 195.212.0.0/16
< X-AS: 2686
< Last-Modified: Sun, 24 Sep 2017 13:28:05 GMT
< ETag: "868647-559ef6976c340"
< Accept-Ranges: bytes
< Content-Length: 8816199
< Content-Type: application/x-gzip
< Connection: Keep-Alive
< Age: 627334
===
Comment 11 Michel Normand 2017-12-20 16:56:07 UTC
Created attachment 753900 [details]
manual_curl_same_host.txt

(In reply to Michael Andres from comment #9)
> [CUT] ...
> - Did you manually download  _on_the_machine_  that shows the error? Would
> be good to know which content `curl -v
> http://download.opensuse.org/.../packages.gz` delivers on this machine.
> (switching to the text console before canceling YAST)
> 
> - We're sure there's no fake download.opensuse.org entry in /etc/hosts which
> redirects to some dummy server? 
> 
> - There's no (transparent) proxy in use which could deliver old data?
> 
> [CUT] ...

If I manually call curl -v as per attached manual_curl_same_host.txt
on same host from where I started the openQA test.
then the data confirms the download has the expected timestamp:
===
* Connected to download.opensuse.org (195.135.221.134) port 80 (#0)
> GET /ports/ppc/factory/repo/oss/suse/setup/descr/packages.gz HTTP/1.1
> User-Agent: curl/7.37.0
> Host: download.opensuse.org
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Wed, 20 Dec 2017 16:02:52 GMT
* Server Apache/2.4.23 (Linux/SUSE) is not blacklisted
< Server: Apache/2.4.23 (Linux/SUSE)
< X-Prefix: 195.212.0.0/16
< X-AS: 2686
< Last-Modified: Tue, 19 Dec 2017 17:25:34 GMT
< ETag: "89db46-560b4c11f7f80"
< Accept-Ranges: bytes
< Content-Length: 9034566
< Content-Type: application/x-gzip
< Connection: Keep-Alive
< Age: 478
===

I do not set by myself any proxy, and do not know how to detect the presence of a transparent proxy.
But I assume that above manual curl test without error is sufficient to say the problem is not related to a proxy.
Comment 12 Michel Normand 2017-12-20 18:42:23 UTC
Created attachment 753912 [details]
manual_curl_same_host_tumbleweed.txt

I identified the source of the problem as difference between "factory" (good as per comment#11 ) and "tumbleweed" as per manual trial of this attachment
===
* Connected to download.opensuse.org (195.135.221.134) port 80 (#0)
> GET /ports/ppc/tumbleweed/repo/oss/suse/setup/descr/packages.gz HTTP/1.1
> User-Agent: curl/7.37.0
> Host: download.opensuse.org
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Wed, 13 Dec 2017 09:12:29 GMT
* Server Apache/2.4.23 (Linux/SUSE) is not blacklisted
< Server: Apache/2.4.23 (Linux/SUSE)
< X-Prefix: 195.212.0.0/16
< X-AS: 2686
< Last-Modified: Sun, 24 Sep 2017 13:28:05 GMT
< ETag: "868647-559ef6976c340"
< Accept-Ranges: bytes
< Content-Length: 8816199
< Content-Type: application/x-gzip
< Connection: Keep-Alive
< Age: 637572
===
Comment 13 Michel Normand 2017-12-20 18:47:05 UTC
So the question:
why do we have differences between two directories ?:

GET /ports/ppc/factory/repo/oss/suse/setup/descr/packages.gz (up to date)
GET /ports/ppc/tumbleweed/repo/oss/suse/setup/descr/packages.gz (obsolete)
Comment 14 Michael Andres 2017-12-21 14:24:21 UTC
Well, I don't see this difference. Neither from within the SUSE network nor on my private machines outside the company network. Nor do I know about any similar report.


Comparing the differences of the two 'curl -v' outputs, you see that not only the Last-Modified tag differs, but also the DATE tag:

>   * Connected to download.opensuse.org (195.135.221.134) port 80 (#0)
>   > GET /ports/ppc/factory/repo/oss/suse/setup/descr/packages.gz HTTP/1.1
> * < Date: Wed, 20 Dec 2017 16:02:52 GMT
>   < Last-Modified: Tue, 19 Dec 2017 17:25:34 GMT
>   < ETag: "89db46-560b4c11f7f80"
>   < Content-Length: 9034566
>   < Age: 478

>   * Connected to download.opensuse.org (195.135.221.134) port 80 (#0)
>   > GET /ports/ppc/tumbleweed/repo/oss/suse/setup/descr/packages.gz HTTP/1.1
> * < Date: Wed, 13 Dec 2017 09:12:29 GMT
>   < Last-Modified: Sun, 24 Sep 2017 13:28:05 GMT
>   < ETag: "868647-559ef6976c340"
>   < Content-Length: 8816199
>   < Age: 637572

This is an old response being returned to you. IMO likely to be some proxies fault.

As 'curl -v .../tumbleweed/.../packages.gz' seems to reliably retrieve the old data, can you try this on some other host in the same network, and also on a host outside the QA or company network? 

Maybe ask your sysadmins whether there is a transparent proxy in use and whether they can check their logs. 

-----
If it was an internal problem at download.opensuse.org, then I'd expect it to be world wide visible and not just on your machine. Anyway you can open a ticket by writing to admin@opensuse.org, asking them to check whether this could be a sever side problem.
(https://en.opensuse.org/openSUSE:Communication_channels#Contact_Info)
Comment 15 Michael Andres 2018-01-11 12:28:18 UTC
Did you solve the puzzle?
Comment 16 Michel Normand 2018-01-11 14:55:08 UTC
(In reply to Michael Andres from comment #15)
> Did you solve the puzzle?

I did not made any change but today's trial did not reported anymore the failure, the dates and sizes are the same between factory and tumbleweed urls:
===
[michel@abanc:~]
$tmplog=/tmp/curl_out.log; echo "$(date)" >$tmplog; for xx in factory tumbleweed; do echo "=== curl for $xx:">>$tmplog; curl -v -o /tmp/packages.$xx.gz http://download.opensuse.org/ports/ppc/$xx/repo/oss/suse/setup/descr/packages.gz >>$tmplog 2>&1; done; grep -E '===|Date|GET|Last|Length' $tmplog
=== curl for factory:
> GET /ports/ppc/factory/repo/oss/suse/setup/descr/packages.gz HTTP/1.1
< Date: Thu, 11 Jan 2018 13:49:19 GMT
< Last-Modified: Mon, 08 Jan 2018 03:22:15 GMT
< Content-Length: 8855217
=== curl for tumbleweed:
> GET /ports/ppc/tumbleweed/repo/oss/suse/setup/descr/packages.gz HTTP/1.1
< Date: Thu, 11 Jan 2018 13:49:26 GMT
< Last-Modified: Mon, 08 Jan 2018 03:22:15 GMT
< Content-Length: 8855217
=== 

I am closing this bug as "worksforme" but I do not understand what was really the cause of previous failure.