EPEL-6 buildrun on RHEL7b1

One of the big challenges we had while building CentOS-6 was to do with scaling the builds. While resources existed, I was unable to get more than 3 ( and in some cases, like perl modules that build with -j1, upto 5 ) concurrent builds. I’ve been quite keen to solve that problem and a bunch of changes, locking in the pre-chroot-build and locking-post-build code had me thinking we could get upto somewhere near the 32 concurrent builds mark.

Based on some very rudimentary maths, 32 concurrent builds will allow us to build/rebuild the distro in just under a day, including the tests, the media and the staging process. In other words, changes to metadata in the build environment would not slow things down for more than a day or so.

---

Because el7b1 itself isnt quite ‘done’ yet, I cant use that as a benchmark – so it was EPEL6 built against the el7b1 content as released upstream; You can read more about that here : http://www.karan.org/blog/2014/01/02/an-epel-build-in-rhel7b1/ and I really don’t recommend people use that content for anything other than academic purposes ( figure out what broke and why …. maybe use some of that on their el7b1 test installs etc ).

The real outcome was that I still cant scale beyond fifteen concurrent workers, without losing the ability to chain builds through; or not use just-built content in subsequent builds. The EPEL6 churn took just over a day, but this was just for the source -> binary conversion ( where and when it did build ); extrapolating back from there it means we should be able to build el7b1 in just over a day and a half, once everything works and we know that the metadata around the builds is good – were not there yet, but getting close.

Given that the number of long running builds has reduced from el6 to el7, the drop in numbers from thirty two to fifteen concurrent builds isn’t that much of a problem – plus, given that we have a much more open process, with the potential for a lot more people to get involved, we should have the environment issues resolved faster.

Continue reading...