Used homebuilts can often be found at attractive prices, but purchasers may not anticipate the operations and handling quirks of an Experimental aircraft. Of these four Fly Babys, three are owned by persons other than the builder.
In May 2012, the NTSB released the results of its study of Experimental/Amateur-Built aircraft accident rates. The report received extensive coverage in the aviation media. Reaction in the homebuilding community was intense, and there was confusion about the source of the data and the basis for the recommended changes. There was little explanation for the reasoning behind the published results.
The reason for the confusion was simple: The NTSB press release was the executive summary, not the report itself. A few weeks later, the full report did get released, a PDF file 169 pages long. The full-report release received little press coverage, and that’s a pity. There’s good information in it, and it does provide the basis for the NTSB’s recommendations.
Process Summary
The report is based mostly on statistical analyses of the NTSB accident database. For any such analysis, the process is the key factor. “Garbage in, garbage out” is one aspect, but even good data can be turned into garbage if the procedures are faulty. A gem cutter can shatter a valuable diamond into worthless dust if he hits it wrong.
Here, the assessment is simple: The NTSB hit it right.
The report’s authors noted problems in the past regarding the inclusion of non-Amateur-Built (AB) aircraft in accident studies, and they made a concerted effort to ensure only AB aircraft were included. They worked with the FAA to validate the certification status of the aircraft included in the report and excluded Special Light Sport (SLSA), Experimental Light Sport (ELSA), Experimental Exhibition and similar non-homebuilt aircraft. Appendix A of the report summarizes the process and notes how the NTSB records themselves are often in error.
Sources were cited (including KITPLANES articles by former Editor Marc Cook and myself), data sources were provided, assumptions were stated and potential errors were discussed. For the portion that went into 2011 statistics, the NTSB provided the accident number for every AB event used in the analysis. NTSB staffers visited kit manufacturers and reviewed government and industry training resources to gain insight into the operation of our sport. It was truly a professional product.
However (and you knew that word was coming), this doesn’t mean that the conclusions the NTSB reached are inarguable. Some may still disagree with their assumptions or how various aspects of the data were assembled.
Engine Issues
Figure 1 shows the top 10 causes for homebuilt accidents from 2001 to 2010. The number one cause of accidents, the NTSB says, is issues affecting the engine. It edged out the “Pilot Loss of Control” category, with 23.2% of the accidents. But when I checked this category against my own database, I got a significantly lower result.
Figure 1: An “Occurrence” is an event that happens as part of an accident. It is not the same as “Probable Cause,” because a given accident can exhibit multiple occurrence types.
The difference is apparently in how the NTSB and I differ on cases where engine power was lost for unknown reasons. In more than 10% of homebuilt accidents, the accident sequence began with a loss of engine power, but the NTSB investigator was unable to determine the cause. In some cases, the engine was too badly damaged and no evidence remained. But often, it is still sufficiently intact. In some instances, they’re even able to restart the engine after the accident.
What actually happened? Some may have been subtle mechanical issues in the engine itself. But more were probably due to transient factors such as carburetor icing or vapor lock, a fuel-system problem where the evidence is destroyed.
In my opinion, counting the undetermined cases in the “Powerplant” failure category was not an ideal approach. These events occur much less often in production-type aircraft, so the results for “System/Component (Powerplant)” between the E-AB and non-E-AB graphs in Figure 1 would be much closer and place it much lower in the top 10.
The NTSB found that powerplant problems occurred more often than pilot loss of control.
Pilot Error and Flight-Test Planning
Referring back to Figure 1, E-AB aircraft have a rate of “Loss of Control in Flight” almost twice as high as the non-E-AB world. The “Loss of Control in Flight” category for fatal accidents stood out even more. The NTSB figures showed that 43.9% of fatal homebuilt accidents involved the pilot losing control of the aircraft.
This is an obvious area for improvement. One of the NTSB’s recommendations from this study was to encourage owners, builders and pilots of E-AB aircraft to complete flight-test training. About a third of kit builders did not receive transition training, and more than 60% of plansbuilt owners did not. The NTSB also wants to see more formal flight-test plans. The report references the EAA Homebuilt Owners’ survey performed in 2011. Survey respondents had been asked about the level of detail in their flight-test plans. Only about three out of eight builders (37%) claimed to have used a detailed test plan.
Coupled with the NTSB’s finding that 9% of homebuilt accidents occur within the first 10 hours after completion and the rate of fatal accidents due to “Loss of Control,” the basis for the NTSB’s recommendation is clear.
However, I must quibble on one aspect of the NTSB’s justification. The NTSB compares the FAA’s lack of requirements for detailed flight-test plans with the requirements of the United Kingdom. The U.K. demands submission of a detailed test plan, approval of the flight-test pilot and generation of a flight manual for the aircraft prior to granting permission for the test period to begin.
However, the comparison lacks one important item: There’s no reference to the relative rate of “Loss of Control” accidents in the U.K. versus the U.S. Without this insight, there’s no way of telling how much of a difference more stringent requirements would make.
The NTSB analysts took special care to include only Experimental/Amateur-Built aircraft such as this RV-6A, and exclude aircraft in other Experimental categories, like the Yaks in the background.
Aircraft Flight Manuals
In addition to its emphasis on adequate flight-test plans, the report also makes a strong push for generating accurate aircraft flight manuals. This is in reaction to the higher accident rate among second or later owners of completed homebuilts. In 2011, for instance, there were more accidents during the first flight by purchasers than by builders. Original builders often take specialized training to prepare for their aircraft’s first flight, but purchasers can be caught unaware by unusual flight characteristics or system operating quirks.
The NTSB feels that the generation of detailed, accurate aircraft flight manuals would reduce this problem. This apparently drove the NTSB to one of its more unfortunate recommendations: that the FAA “…require the review and acceptance of aircraft operating limitations and supporting documentation as a condition of registration or re-registration [emphasis added] of an Experimental/Amateur-Built aircraft.”
This would essentially eliminate permanent airworthiness certificates for homebuilts. The purchaser of a flying homebuilt would stand the risk that the FAA might not approve the aircraft for continued operation. Also, as the FAA now requires regular re-registration of all N-numbered aircraft, every homebuilt would thus have to undergo an FAA review every three years.
I’m guessing the report writers were focused on ensuring that detailed flight manuals were available to purchasers of used homebuilts and didn’t consider the implications for existing aircraft.
The flight-manual issue is a thorny one. The big kit manufacturers will easily be able to assemble templates for their customers, but what about the plansbuilt folks? For some, writing an aircraft flight manual offers a much more difficult problem than the building of an aircraft. Will Joe Smith’s Mini-Max aircraft flight manual be held to the same standards as that of a Lancair IV-P? If so, the requirement would be truly unfair to the plansbuilt segment of the sport.
Fuel-System Testing
In addition to the flight-test-planning comparison with the U.K., the NTSB makes a similar reference to Canada’s fuel-system testing requirements, again, with no comparison of accident rates. This is related to another of their major recommendations: that the FAA institute a requirement for formal fuel-system testing as a condition for receiving an airworthiness certificate.
This type of fuel-flow test for homebuilts is suggested by FAA Advisory Circular 90-89A, but is required by Transport Canada. The NTSB report recommended that formal documentation of the test be required and approved by the FAA.
This is a tough recommendation to argue against. Of course, fuel-system functional testing is a good idea. Every homebuilder reference, including FAA Advisory Circular 90-89A, recommends a fuel-flow test to ensure that the fuel system can deliver at least 1.5 times the full-throttle fuel flow in climb attitude. This is the same test Transport Canada requires.
The question is whether there are enough people who don’t perform such a test to justify the institutional overhead and overall hassle associated with formally documented testing.
My own homebuilt accident database shows about 94 accidents involving fuel-system problems between 2001 and 2010. Indeed, among hardware-induced accidents, fuel-system issues are second only to powerplant problems. Eleven of these accidents occurred on the aircraft’s first flight—a pretty high proportion.
Figure 2 shows the breakdown of causes for homebuilts accidents during first flight, the rest of the 40-hour test period and beyond the test period. Blockage issues (including filters, vents or lines) and fuel-line problems constitute a large proportion on the first flight—more in that first hour than in the 39 hours that follow!
Figure 2: The single hours of homebuilts’ first flights show more accidents related to blocked or leaking fuel systems than the remaining 39 of the Phase I flight-test period.
Would FAA-mandated fuel-flow testing have prevented these? Possibly. But we’re talking about just a single first-flight accident per year, on average. And of those 11 first-flight accidents in the 10-year time frame, only one resulted in a fatality.
Any aircraft accident is a tragedy, regardless of the degree of injury. But instituting a requirement for formal reporting of homebuilt fuel-flow test results would be of little benefit. Again, all builders should run such tests. It’s just that the additional paperwork won’t produce much safety improvement.
They’re Out to Get Us! (Or Are They?)
Objections are often raised when outsiders are critical of E-AB aircraft safety. Some folks claim, “They’re biased toward the big manufacturers!” or, “They don’t understand homebuilts,” or even, “They don’t like our freedom and are trying to wreck the sport!”
The NTSB report recommended updates for several Advisory Circulars dealing with homebuilt aircraft.
That’s not to say they’re always wrong—after all, even paranoid people have enemies. But the analysis for the NTSB’s “The Safety of Experimental/Amateur-Built Aircraft” report was performed scientifically, with full disclosure of data and processes. It’s a quality product, making some good points with few, if any, signs of anti-homebuilding bias. The recommended re-registration documentation review is a major area of concern, but I believe the NTSB didn’t consider the implications in light of the FAA’s recent requirement for regular re-registration of all aircraft.
We may quibble with some of the assumptions used or the magnitude of the problems uncovered, but the basic findings of the report are hard to argue with: Homebuilts do have a higher accident rate than production aircraft. This rate could be reduced by more detailed flight-test planning and the creation of accurate and complete flight manuals. And purchasers of used homebuilts are not being prepared properly for transition into aircraft that sometimes exhibit unusual flight characteristics.
The report recommended that the FAA and EAA help create a coalition of kit manufacturers and other groups to develop transition-training resources and encourage their use.
A better safety record would benefit us all. We don’t have to accept all of the recommendations, but the NTSB has given us a batch of good, solid advice.
The full NTSB report can be found at http://go.usa.gov/fmf or at www.wanttaja.com/ntsb.pdf.
Summary of Recommendations
To the Federal Aviation Administration:
• Define aircraft fuel-system functional test procedures and require applicants for an airworthiness certificate to conduct that test and submit a report of the results.
• Require applicants for an airworthiness certificate to submit, for FAA acceptance, a flight-test plan that will: 1) ensure the aircraft has been adequately tested, and 2) produce flight-test data to develop an accurate and complete aircraft flight manual and to establish emergency procedures.
• Encourage owners, builders and pilots to complete a flight-test training regimen.
The NTSB report contains numerous recommendations, but remember: The NTSB doesn’t have regulatory authority. The FAA is responsible for implementing, modifying or rejecting the NTSB’s suggestions.
• Clarify circumstances in which a second qualified pilot could be authorized to assist in the perfor- mance of Phase I flight tests.
• Require the FAA review and acceptance of the completed test-plan documents and aircraft
flight manual.
• Revise AC 90-89A, Amateur-Built Aircraft and Ultralight Flight Testing Handbook, to include guidance for the use of recorded flight data for the purposes of flight testing and maintaining continued airworthiness of Experimental aircraft.
• Include provisions for the use of electronic data recordings from electronic flight displays, engine instruments or other recording devices in support of Phase I flight testing.
• Publish guidance for the issuance of a Letter of Deviation Authority (LODA) to conduct flight instruction in an Experimental aircraft.
• Create a coalition of kit manufacturers, type clubs and pilot and owner groups to develop transition-training resources and encourage both builders and purchasers to complete the training.
• Require the review and acceptance of aircraft operating limitations as a condition of registration or re-registration of an Experimental/Amateur-Built aircraft.
• Revise guidance or regulations, as necessary, to include provisions for modifying the operating limitations of aircraft previously certificated as Experimental.
• Revise the Civil Aircraft Registry database to include a means of identifying aircraft manufacturer, make, model and series to unambiguously identify the aircraft kit or plans design, as well as the builder of the aircraft.
To the Experimental Aircraft Association:
• Encourage owners, builders and pilots to complete a flight-test training regimen.
• Develop standards for the recording of data in electronic flight displays, engine instruments or other recording devices to be used in support of flight testing.
• Publish a repository of information regarding holders of Letters of Deviation Authority (LODA) to conduct flight instruction in Experimental aircraft.
• Create a coalition of kit manufacturers, type clubs and pilot and owner groups to develop transition-training resources and encourage both builders and purchasers to complete the training.
—R.W.
