Bugzilla – Bug 1077218
llvm5: build often stalls obs workers
Last modified: 2018-05-25 09:19:34 UTC
Quite often, I see llvm5 builds failing due to the workers apparently stalling
This is then detected by OBS (after 8 hours of inactivity) and the job being marked as 'failed'
I've seen the stalls in various locations, like for example:
[ 4817s] -- Installing: /home/abuild/rpmbuild/BUILDROOT/llvm5-5.0.1-3.2.x86_64/usr/bin/llvm-pdbutil
[ 4817s] -- Installing: /home/abuild/rpmbuild/BUILDROOT/llvm5-5.0.1-3.2.x86_64/usr/bin/llvm-readobj
[ 4817s] Creating llvm-readelf
[ 4817s] -- Installing: /home/abuild/rpmbuild/BUILDROOT/llvm5-5.0.1-3.2.x86_64/usr/bin/llvm-rtdyld
[ 4817s] -- Installing: /home/abuild/rpmbuild/BUILDROOT/llvm5-5.0.1-3.2.x86_64/usr/lib64/libLLVM.so.5.0.1
(build has been going on for '4 hours')
often it stalls during rpm write process, debuginfo extraction
Bad enough that llvm5 takes that long to build - but even worse if it takes more than one attempt to get going
The stalls seem to correlate to the lamb77 and lamb78 workers; cumulus seem not to suffer from it
They had an out-of-disk-space situation, I reduced them from 3 to 2 parallel instances now.
Does it mean LLVM needed more than the 50GB of disk space required by _constraints? (I.e. something I should fix in llvm.) Or did the worker actually not have the required space?
Note that the llvm disk usage situation will get a bit better soon since I'll have to get rid of static libraries because of bug 1065464.
Bug just got reassigned to me... Many optimizations were done in llvm build since this bug was opened. The build is now faster, needs less disk space and seems to be no longer stalling on OBS workers. So closing this as fixed.