JPRT: A Build & Test System
- by kto
DRAFT
A while back I did a little blogging on a system called JPRT,
the hardware used
and a summary on my
java.net weblog.
This is an update on the JPRT system.
JPRT ("JDK Putback Reliablity Testing", but ignore what the letters
stand for, I change what they
mean every day, just to annoy people :\^)
is a build and test system for the JDK, or any source base
that has been configured for JPRT.
As I mentioned in the above blog, JPRT is a major modification to a
system called PRT that the HotSpot VM development
team has been using for
many years, very successfully I might add.
Keeping the source base always buildable
and reliable is the first step in the 12 steps of dealing
with your product quality... or was the 12 steps from Alcoholics Anonymous... oh well, anyway, it's the first of many steps. ;\^)
Internally when we make changes to any part of the JDK, there
are certain procedures we are required to perform prior to any
putback or commit of the changes.
The procedures often vary from team to team, depending on many
factors, such as whether native code is changed, or if the change
could impact other areas of the JDK.
But a common requirement is a verification that the source base
with the changes (and merged with the very latest source base) will
build on many of not all 8 platforms,
and a full 'from scratch' build, not an incremental
build, which can hide full build problems.
The testing needed varies, depending on what has been changed.
Anyone that was worked on a project where multiple engineers or
groups are submitting changes to a shared source base
knows how disruptive a 'bad commit' can be on everyone.
How many times have you heard:
"So And So made a bunch of changes and now I can't build!".
But multiply the number of platforms by 8, and make all the platforms
old and antiquated OS versions with bizarre system setup
requirements and you have a pretty complicated situation
(see
http://download.java.net/jdk6/docs/build/README-builds.html).
We don't tolerate bad commits, but our enforcement is somewhat
lacking, usually it's an 'after the fact' correction.
Luckily the Source Code Management system we use (another
antique called TeamWare) allows for a tree of repositories
and 'bad commits' are usually isolated to a small team.
Punishment to date has been pretty drastic,
the Queen of Hearts in 'Alice in Wonderland' said
'Off With Their Heads', well trust me, you don't want to
be the engineer doing a 'bad commit' to the JDK.
With JPRT, hopefully this will become
a thing of the past, not that we have had many 'bad commits' to
the master source base,
in general
the teams doing the integrations know how important their jobs are
and they rarely make 'bad commits'.
So for these JDK integrators, maybe what JPRT does is keep them from
chewing their finger nails at night. ;\^)
Over the years each of the teams have accumulated sets of machines
they use for building, or they use some of the shared machines
available to all of us. But the hunt for build machines is just
part of the job, or has been. And although the issues with
consistency of the build machines hasn't been a horrible problem,
often you never know if the Solaris build machine you are using
has all the right patches, or if the Linux machine has the right
service pack, or if the Windows machine has it's latest updates.
Hopefully the JPRT system can solve this problem.
When we ship the binary JDK bits, it is SO very important that the
build machines are correct, and we know how difficult it is
to get them setup.
Sure, if you need to debug a JDK problem that only shows up on
Windows XP or Solaris 9, you'll still need to hunt down a
machine, but not as a regular everyday occurance.
I'm a big fan of a regular nightly build and test system,
constantly verifying that a source base builds and tests out.
There are many examples of automated build/tests, some that trigger
on any change to the source base, some that just run every night.
Some provide a protection gateway to the 'golden' source base
which only gets changes that the nightly process has verified
are good.
The JPRT (and PRT) system is meant to guard the source base
before anything is sent to it, guarding all source bases
from the evil developer, well maybe 'evil' isn't the right word,
I haven't met many 'evil' developers, more like 'error prone'
developers. ;\^) Humm, come to think about it, I may be one
from time to time. :\^{
But the point is that by spreading the build up over a set of
machines, and getting the turnaround down to under an hour,
it becomes realistic to completely build on all platforms and
test it, on every putback. We have the technology, we can build
and rebuild and rebuild, and it will be better than it was before,
ha ha... Anybody remember the Six Million Dollar Man?
Man, I gotta get out more often..
Anyway, now the nightly build and test can become a 'fetch the latest
JPRT build bits' and start extensive testing (the testing
not done by JPRT, or the platforms not tested by JPRT).
Is it Open Source? No, not yet.
Would you like to be? Let me know.
Or is it more important that you have the ability
to use such a system for
JDK changes?
So enough blabbering on about this JPRT system, tell me what you
think.
And let me know if you want to hear more about it or not.
Stay tuned for the next episode, same Bloody Bat time,
same Bloody Bat channel. ;\^)
-kto
