Bug 1161106 - rpmlint: frequently fails to check rust on i586
rpmlint: frequently fails to check rust on i586
Status: RESOLVED INVALID
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Other
Current
Other Other
: P2 - High : Normal (vote)
: ---
Assigned To: Dirk Mueller
E-mail List
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2020-01-16 14:34 UTC by Dominique Leuenberger
Modified: 2020-01-17 11:07 UTC (History)
8 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dominique Leuenberger 2020-01-16 14:34:55 UTC
so far I've mainly seen this on the rust package on i586, but, rpmlint, since the last larger changes (end of December) frequently causes failures when checking rust:

[ 8192s] RPMLINT report:
[ 8192s] ===============
[ 8283s] 11 packages and 0 specfiles checked; 3 errors, 9 warnings.
[ 8283s] Traceback (most recent call last):
[ 8283s]   File "rpmlint.py", line 378, in <module>
[ 8283s]   File "rpmlint.py", line 166, in main
[ 8283s]   File "rpmlint.py", line 224, in runChecks
[ 8283s]   File "AbstractCheck.py", line 50, in check
[ 8283s]   File "SpecCheck.py", line 155, in check_source
[ 8283s]   File "SpecCheck.py", line 201, in check_spec
[ 8283s]   File "Pkg.py", line 208, in readlines
[ 8283s] FileNotFoundError: [Errno 2] No such file or directory: '/tmp/rpmlint.rust-1.40.0-5.2.src.rpm.7vwdq64_/rust.spec'
Comment 1 Dominique Leuenberger 2020-01-16 14:35:42 UTC
This keeps on failing in the various stagings, and is thus pretty annoying
Comment 2 Matthias Gerstner 2020-01-17 10:35:07 UTC
I gather that this is only happening sporadically? This will be hard to track
down then ...

Have you only seen this with rust or also with other RPMs?

That the spec file in the temporary directory disappears or never arrives
there could mean that some other component breaks things in parallel, e.g. by
removing files in /tmp, or by mounting something over /tmp. Or the extraction
of the RPM content fails and isn't recognized by rpmlint. An our of space
error could also be the reason.

Ah I'm just seeing that size might actually be the problem. The src
RPM for rust is 2.1 GB in size. So probably your /tmp file system is too small
and rpmlint isn't catching ENOSPC errors correctly?
Comment 3 Dominique Leuenberger 2020-01-17 10:55:01 UTC
(In reply to Matthias Gerstner from comment #2)
> I gather that this is only happening sporadically? This will be hard to track
> down then ...

sporadically - but often enough to be annoying (especially, as mentioned, rust is not just attempted to be rebuilt in 5 minutes)

> Have you only seen this with rust or also with other RPMs?

So far I've only observed it with rust - and only on i586 builds

> That the spec file in the temporary directory disappears or never arrives
> there could mean that some other component breaks things in parallel, e.g. by
> removing files in /tmp, or by mounting something over /tmp. Or the extraction
> of the RPM content fails and isn't recognized by rpmlint. An our of space
> error could also be the reason.
> 
> Ah I'm just seeing that size might actually be the problem. The src
> RPM for rust is 2.1 GB in size. So probably your /tmp file system is too
> small
> and rpmlint isn't catching ENOSPC errors correctly?

The jobhistory for rust/i586 in openSUSE:Factory:Staging:J looks like this:

2019-12-20 19:21:46  rust   new build        succeeded    2h 55m 51s   lamb52:8        
2019-12-28 01:57:44  rust   meta change      succeeded    3h 55m 10s   lamb60:4        
2020-01-13 23:06:41  rust   source change    failed       3h  0m 44s   lamb52:4        
2020-01-14 03:46:45  rust   new build        failed       3h 40m 34s   lamb21:2        
2020-01-14 06:41:00  rust   new build        failed       2h 35m 29s   lamb53:8        
2020-01-14 12:36:27  rust   new build        failed       2h 58m 13s   lamb56:6        
2020-01-14 18:05:15  rust   new build        failed       3h 13m 19s   lamb58:1        
2020-01-14 22:02:30  rust   new build        succeeded    1h 59m 44s   lamb63:1        
2020-01-15 00:56:40  rust   meta change      succeeded    2h 48m 54s   build71:1       
2020-01-16 05:18:02  rust   meta change      failed       2h 57m  1s   lamb55:8        
2020-01-16 08:25:40  rust   new build        failed       2h 20m 11s   lamb23:11       
2020-01-16 10:50:57  rust   new build        failed       2h 15m 51s   lamb57:4        
2020-01-16 13:17:32  rust   new build        failed       2h  9m 17s   lamb25:11       
2020-01-16 16:50:58  rust   new build        failed       3h 14m 11s   lamb57:2        
2020-01-16 19:52:34  rust   new build        failed       2h 38m  5s   lamb21:11       
2020-01-17 01:00:36  rust   new build        failed       3h  0m 27s   lamb25:5        

Looking at the workers with successful vs unsuccessfil builds, disk space constraints indeed can be an issue: the failing one all have 16GiB of storage - the succeeding ones have significantly more.

So, for rpmlint, a slightly 'nicer' error message would be beneficial (but I'll try to workaround the issue in rust build directly)
Comment 4 Matthias Gerstner 2020-01-17 10:57:37 UTC
(In reply to matthias.gerstner@suse.com from comment #2)
> Ah I'm just seeing that size might actually be the problem. The src
> RPM for rust is 2.1 GB in size. So probably your /tmp file system is too small
> and rpmlint isn't catching ENOSPC errors correctly?

And indeed I can reproduce the issue by mounting a small tmpfs on /tmp and
running rpmlint on the rust source RPM. strace shows the following call
pattern:

```
strace -e execve,write,exit -f rpmlint rust-1.40.0-250.1.src.rpm |& grep -e ENOSPC -e execve -e exit
execve("/usr/bin/rpmlint", ["rpmlint", "rust-1.40.0-250.1.src.rpm"], 0x7ffc1d7b7e40 /* 63 vars */) = 0
[pid  4988] execve("/sbin/ldconfig", ["/sbin/ldconfig", "-p"], 0x7fac24a9dbd0 /* 2 vars */) = 0
[pid  4988] +++ exited with 0 +++
[pid  4989] execve("/bin/sh", ["/bin/sh", "-c", "rpm2cpio rust-1.40.0-250.1.src.r"...], 0x5617f3abb1d0 /* 64 vars */) = 0
[pid  4990] execve("/usr/bin/rpm2cpio", ["rpm2cpio", "rust-1.40.0-250.1.src.rpm"], 0x55cefd8c0190 /* 64 vars */) = 0
[pid  4992] execve("/usr/bin/cpio", ["cpio", "-id"], 0x55cefd8c1df0 /* 64 vars */) = 0
[pid  4992] write(3, "\320\364\301\27]\206]\363ZT\322\257\30\374\204\324\342\326ub\3164\367\215\315\360\360\10q\230\372\35"..., 512) = -1 ENOSPC (No space left on device)
[pid  4992] +++ exited with 2 +++
[pid  4991] +++ exited with 2 +++
[pid  4994] execve("/usr/bin/chmod", ["chmod", "-R", "+rX", "/tmp/rpmlint.rust-1.40.0-250.1.s"...], 0x55cefd8c0190 /* 64 vars */) = 0
[pid  4994] +++ exited with 0 +++
[pid  4989] +++ exited with 0 +++
[pid  4995] execve("/sbin/gzip", ["gzip", "-dcf", "/tmp/rpmlint.rust-1.40.0-250.1.s"...], 0x5617f3b23270 /* 65 vars */) = -1 ENOENT (No such file or directory)
[pid  4995] execve("/usr/sbin/gzip", ["gzip", "-dcf", "/tmp/rpmlint.rust-1.40.0-250.1.s"...], 0x5617f3b23270 /* 65 vars */) = -1 ENOENT (No such file or directory)
[pid  4995] execve("/usr/local/sbin/gzip", ["gzip", "-dcf", "/tmp/rpmlint.rust-1.40.0-250.1.s"...], 0x5617f3b23270 /* 65 vars */) = -1 ENOENT (No such file or directory)
[pid  4995] execve("/root/bin/gzip", ["gzip", "-dcf", "/tmp/rpmlint.rust-1.40.0-250.1.s"...], 0x5617f3b23270 /* 65 vars */) = -1 ENOENT (No such file or directory)
[pid  4995] execve("/usr/local/bin/gzip", ["gzip", "-dcf", "/tmp/rpmlint.rust-1.40.0-250.1.s"...], 0x5617f3b23270 /* 65 vars */) = -1 ENOENT (No such file or directory)
[pid  4995] execve("/usr/bin/gzip", ["gzip", "-dcf", "/tmp/rpmlint.rust-1.40.0-250.1.s"...], 0x5617f3b23270 /* 65 vars */) = 0
[pid  4995] +++ exited with 1 +++
[pid  4996] execve("/sbin/rm", ["rm", "-rf", "/tmp/rpmlint.rust-1.40.0-250.1.s"...], 0x5617f3b23270 /* 65 vars */) = -1 ENOENT (No such file or directory)
[pid  4996] execve("/usr/sbin/rm", ["rm", "-rf", "/tmp/rpmlint.rust-1.40.0-250.1.s"...], 0x5617f3b23270 /* 65 vars */) = -1 ENOENT (No such file or directory)
[pid  4996] execve("/usr/local/sbin/rm", ["rm", "-rf", "/tmp/rpmlint.rust-1.40.0-250.1.s"...], 0x5617f3b23270 /* 65 vars */) = -1 ENOENT (No such file or directory)
[pid  4996] execve("/root/bin/rm", ["rm", "-rf", "/tmp/rpmlint.rust-1.40.0-250.1.s"...], 0x5617f3b23270 /* 65 vars */) = -1 ENOENT (No such file or directory)
[pid  4996] execve("/usr/local/bin/rm", ["rm", "-rf", "/tmp/rpmlint.rust-1.40.0-250.1.s"...], 0x5617f3b23270 /* 65 vars */) = -1 ENOENT (No such file or directory)
[pid  4996] execve("/usr/bin/rm", ["rm", "-rf", "/tmp/rpmlint.rust-1.40.0-250.1.s"...], 0x5617f3b23270 /* 65 vars */) = 0
[pid  4996] +++ exited with 0 +++
```

So `cpio` sees the ENOSPC, but the `rpm2cpio` process receives a SIGPIPE and
dies this way. The `/bin/sh -c rpm2pcio ...` process exists with zero exit
status, thus rpmlint doesn't recognize an error and only a follow-up error
occurs.

This could be something for an upstream error report. For our staging it
simply means that /tmp needs to be enlarged, I think.
Comment 5 Dominique Leuenberger 2020-01-17 11:07:58 UTC
I doubt the workers make such a complex disk layout - they only mount / (which includes /tmp)

But I found a possibly easy fix for the rust.src.rpm that should make it much smaller

Thanks for the pointers. Let's close this bug report. I'll try to formulate a bug for rpmlint upstream about passing on error messages from subprocesses