Naturally, some might wonder how this change is possible. Some might wonder whether we have made the exams shorter by making them easier — that perhaps we have eliminated a lot of coverage. Others might wonder whether we have deliberately made them more difficult by forcing people to do as much or more in a shorter period of time. Neither of these conjectures apply.

The people who would pass these exams on April 30 would probably pass on May 1. The people who would not pass on April 30 would probably meet the same fate on May 1.

We have shortened the testing time by consolidating the two separate sections of the earlier format and by using time more efficiently. We have found, for example, that candidates who passed Section II of these exams seldom needed as much time as we allocated in Section I. Furthermore, virtually all candidates are capable of installing Red Hat Enterprise Linux. Consequently, instead of requiring a bare-metal installation, we will provide pre-installed systems, which will save 20-25 minutes.

We have also made a few minor content changes, but our analysis suggests that these changes can be made without significantly changing outcomes for candidates.

Timing is not the only change. The 100K+ candidates of the RHCT and RHCE exams will recall getting a printed exam document that listed the tasks they must perform. The bare-metal installation required under earlier versions of the exam made this a necessity: how else would you know what to install? Under the new format, the tasks are presented electronically, thus eliminating paper.

Eliminating paper has led to other advances. The electronic format means that candidates can choose their preferred language, regardless of where they take the exam. Incidentally, the exams are now available in 5 new languages (Simplified Chinese, Traditional Chinese, Dutch, Italian, and Turkish) in addition to the 7 previously available (English, Japanese, French, Korean, Russian, Spanish, and Portuguese.) Few IT certification exams can claim this level of internationalization.

Red Hat’s ten-year commitment to performance-based testing has built a dedicated community of Red Hat certified IT professionals. We do not take their dedication lightly and will not make changes to certification programs or exams arbitrarily. A good program must evolve, however, and we believe that these changes will make for a more efficient and effective test. Some may feel more comfortable with the notion that nothing should ever change. I wonder if those same people are using the 0.10 Linux kernel?

In particular, check out the new video with photos and footage of RHCEs from around the world. We will be adding to the video throughout the year to create a scrapbook. Please help us celebrate this milestone for Red Hat and its certified community.

Why are we taking a different approach?

Back in 1999 when Bob Young was CEO of Red Hat, he would pose a question regularly: “Would you buy a car with the hood welded shut?” (The hood is what the British call the “bonnet”.) He would assert that proprietary software is similar to such a car. You cannot see how it works. You cannot investigate problems and no one else besides the software vendor can either. Young’s point was that proprietary software implies a lack of true ownership, insight, and control that we would never tolerate with other goods or services.

Ten years later, I find myself wondering if IT certification programs are a lot like that imaginary car with the hood welded shut. Employers use certifications as a basis for their hiring and management decisions, but typically do so without any knowledge of what these certifications represent. How could they possibly know? Tests are kept confidential.

Programs sometimes provide information about what they test in general terms. They sometimes describe the content areas covered by the certification in general terms. They sometimes report the percentage of test items that cover these areas. They sometimes report who was involved in writing test items. None of this information tells us much about what they actually test. Consequently, there is little choice but to trust these programs and their testing…just as one would have to “trust” a car manufacturer that welds the hoods of its cars shut.

Most programs would point to various statistical measures they calculate and track — psychometrics — and insist that these measures somehow prove they can be trusted with what is unseen under the hood. I would argue that these measures are no more or less useful than being told that the oil level is at Full in a car with its hood welded shut. The information is useful and relevant, but incomplete.

A full oil pan does not tell us whether the engine functions properly. It does not tell us whether it produces a lot of horsepower or just a little. It does not tell us whether the whole car will explode into a fireball after a short drive.

It is time for those of us in IT testing to begin breaking the welds on the hoods of these cars we are producing. The hood is not open yet with the programs my team and I manage at Red Hat. We have followed many exam security conventions for ten years and we cannot responsibly drop them all overnight.

We will, however, proceed thoughtfully and deliberately towards ever-greater transparency. The JBoss Application Administrator Exam Blueprint Survey is just our first step.

The purpose of this survey is to validate and improve our working model of what a qualified JBoss application administrator should be able to do on the job. This model will be used to develop a performance-based certification exam for JBoss application administrators.

The survey began as a definition document — a rough outline of sorts. We then gathered a small group of JBoss specialists from various backgrounds to provide more detail and specificity. We dropped some skills from the list. We added others. After a couple of days we ended up with consensus (mostly!) about what an application administrator should be able to do.

We then solicited feedback from JBoss professionals inside Red Hat to make sure we were on the right track. While we did need to make a few minor changes, the feedback we received supported our model.

The next step is critical. We need many more eyes on this survey. The more perspectives we can get from JBoss pros worldwide, the better this certification will be. Tell us if we are missing something. Validate our work if you think we are on the right track.

We will be doing a lot of our work on this program right out here in the open, where anyone who is interested can contribute their ideas. It is a somewhat different approach to test development, but we believe it is an approach that will net great results. We know far more collectively than any small group of us can in isolation. Help us build a great program.

Please join your fellow federal government RHCEs and RHCTs on March 26th for our inaugural Government RHCE and RHCT Forum.

Satellite Server Product Manager Todd Warner will be talking about Red Hat Network Satellite Server. Gunnar Hellekson, Lead Architect and Marc will also provide an insight into the product roadmap and offer an opportunity for you to provide feedback. Last and possibly least, yours truly will be there to update you on our plans for Red Hat certification and most importantly and get your input as we begin our second decade of the program.

Still not convinced? Okay, how about this: Lunch is provided! It is FREE!

Help me spread the word on this event. I am very excited about the chance for us to meet and hear from RHCTs and RHCEs working in this critical sector.

To register, please visit the registration page.

RHCE and RHCT Federal Forum

When:
March 26, 2009
12:00 – 16:00 EDT

Where:
The Tower Club
8000 Towers Crescent Drive
Vienna, VA 22182

Agenda:
12:00pm – 12:30pm:
Registration

12:30 – 14:00:
Working Lunch
Satellite Server Overview and Best Practices
Todd Warner, Product Manager, Red Hat

14:00 – 15:00:
New Product Roadmap Review & Feedback
Gunnar Hellekson, Lead Architect, Red Hat

15:00pm – 16:00pm:
Certification Forum
Randy Russell, Director of Certification, Red Hat

Advocates of performance testing sometimes assert that item security is much less important for performance items than for multiple choice items. One must still know how to perform the tasks required by a performance item, regardless of whether you know the item beforehand. On the other hand, candidates can memorize cues in a multiple choice item that would later enable them to identify a correct answer even though they may have no idea why the answer is correct.

Some rebut the assertion that item security is less important for performance tests by pointing out that tests sample from a domain and are used to infer people’s knowledge over the entire domain, not just the sample. If items on a performance test are exposed, candidates who prepare only for the exposed tasks could pass the test. Passing may mean they are qualified in test’s sample of the domain, but we could not draw accurate inferences about a candidate’s abilities in the rest of the domain.

What if a performance exam tested most or all of a domain? I would assert that item security becomes progressively less important as the sample size of the domain increases in a performance test. I am reluctant to make claims about other domains, but for the technical skills we test in Red Hat exams, performance testing makes it possible to test a far greater proportion of the domain in a given time than we could ever test using multiple choice. The reason is item efficiency: performance items reveal more information.

An Example
To illustrate this point, consider the simple example of a performance item that directs candidates to configure a web server to run on a system automatically after a reboot and that must be network-accessible to other systems. In practice, such an item should be worded more carefully but I am simplifying for the sake of less technical readers.

Candidates must execute a number of steps to complete this item successfully. First, candidates must install the necessary software packages. There are several ways to do it. The packages have dependencies; that is, in order to install Package B, Package A must be installed first. Two of the methods available for installing packages resolve these dependencies for you. They ensure that all dependencies are satisfied and that packages are installed in the proper order. Another, perfectly legitimate installation method requires that you resolve these dependencies manually.

The packages are stored on a different system. The installation methods that resolve dependencies require some configuration to work because the tools must know where to find all the packages needed to resolve the dependencies. Configuring this capability requires that one know how to edit text files in addition to understanding the changes one must make to the configuration file.

The method requiring one to resolve dependencies manually does not require any configuration. However, one must either know how to specify package locations using a URL or else know how to download packages first, then use the tool to install the packages manually. We have just begun the process of configuring our web server, yet consider how many multiple choice items we would need in order to gauge a candidate’s ability to get this far in the process.

We Are Not Done Yet
Installing the software does not enable it to run. We must configure the web server to run when the system is booted. This is a simple process typically handled by a single command, but once again there are several other ways to do it.

Once the web server is configured to start on boot we are almost done. Many configuration options are possible, including some that are rather specialized and complex. These might not be relevant for the skill level for which we are testing. For our example, let us assume the simplest possible case in which the default configuration is all that is required. If we have installed the web server software and configured it to run automatically, we might think we have done everything necessary. We have not.

In order for the web server to be accessible to other systems, we must also modify the default firewall configuration. Once again, there are a few different ways to do that. Doing it correctly requires some understanding of basic TCP/IP networking principles. This TCP/IP knowledge must be used in conjunction with knowledge of one of the mechanisms available for configuring the firewall. Furthermore, a smart candidate will realize that it is not enough to create an opening in the firewall while the system is up and running. The requirement is that the web server must be available after a reboot, so the opening in the firewall must also be configured to work after a reboot just as we must configure the web server run automatically after a reboot. Oh yeah…one must also be careful not to break access to other networking services by misconfiguring the firewall!

Conclusion
Consider all the information we would have gathered from this single performance item. How many multiple choice items would be required to gather the same information? Consider this, too: all the work necessary to accomplish these tasks can be accomplished in just a few minutes by people who know what they are doing.

This brings me back to the question about sampling from a domain and its relationship to item security. Multiple choice samples rather sparingly. We must present many items in order to gather a meager bit of information. Performance testing samples greedily. We can gather a wealth of information from just a few items. If we are willing to present more than a few, we can gather a whole lot of information — possibly most or even all of the domain being tested. This premise opens the door to a completely different way of thinking about security and testing that we will be exploring in the year ahead.