In the midst of the crisis around the Boeing-737 Max, it still remains a mystery: how the company, renowned for its careful approach to design, made, apparently, childhood mistakes in software development, which led to two disasters with fatalities. Engineers who have worked for the company for many years say development has been complicated by outsourcing some of the work to low-wage contractors.
The flaws in the software could leave planes grounded for another month - US regulators have discovered additional problems this week. The software for the 737-Max series was written at a time when Boeing was laying off experienced engineers and pressured suppliers.
What's more, the American aircraft icon and its subcontractors trusted temporary workers, earning as little as $ 9 an hour, to develop and test their software.
Often, these were workers from countries with undeveloped aircraft manufacturing, namely from India.
"Yesterday's graduates, hired by the Indian software company HCL Technologies Ltd, occupy several rows of desks in the Boeing Field offices in Seattle (officially King County International Airport, at this airport the Boeing company has its hangar and tests aircraft - approx. Transl.)" says Mark Rabin, a former Boeing engineer who worked on the 737-Max aircraft testing team.
HCL coders usually develop according to specifications sent from Boeing. But, according to Rabin, "this is a controversial decision, as it is much less efficient than just letting Boeing engineers write the code." He recalls that "it was often necessary to redo everything several times because the code was written incorrectly."
The support of Indian companies may have brought other benefits as well. Boeing has won several tenders over the past few years to supply military and commercial aircraft to India, such as a $ 22 billion contract for SpiceJet Ltd. The contract includes 100 737-Max 8 aircraft and is the largest order in the history of a traditional Indian airline with Airbus.
According to the findings published on social media, engineers from HCL were involved in the development and testing of software for the PFD (Primary flight display), and employees of another Indian company, Cyient Ltd., were involved in software for instrumentation. designed for flight tests.
Promotional video:
Costly delay
In one of the posts, an HCL employee described his job responsibilities as follows: "I quickly made a crutch to solve the problem in production and not delay the flight tests of the 737-Max (delaying each flight costs Boeing a huge amount)."
Boeing says it did not trust the engineers at HCL and Cyient to develop the Maneuvering Characteristics Augmentation System (MCAS), which is associated with the crashes of Lion Air flight JT-610 near Jakarta in October 2018 and Ethiopian Airlines flight ET302 near Addis Ababa in March 2019. Also, according to Boeing, none of these companies are associated with a problem discovered after the disasters - the warning light in the cockpit that is not working for most buyers.
“Boeing has many years of experience working with suppliers and partners around the world,” says a company spokesman. “Our main goal is to always make sure that our products are safe, of the highest quality and are performed in accordance with all the rules.”
In turn, HCL in an official statement states that “has a strong and longstanding business relationship with Boeing and is proud of the work that the company has done for its customers. However, HCL does not comment on what kind of work it was. HCL is in no way related to current problems with the 737 Max.”
Recent tests on the emulator by the US Federal Aviation Administration (FAA) reveal that the software problems lie at a deeper level. The company's stock fell this week after regulators discovered an issue with the chip, which lags behind in critical responses if it is overwhelmed with requests.
Development of the 737 Max began 8 years ago, and the engineers working on it complained of pressure from managers. Demands have been made to limit changes that potentially create additional costs.
“Boeing has done everything you can, everything you can imagine, to cut costs, including relocating development from Puget Sound, because it costs too expensive,”says Rick Ludtke, a former flight test engineer who was fired in 2017. “This is understandable if you look at the situation from a business point of view. Gradually, over time, it turned out that this weakened the design ability of Puget Sound engineers.”
Rabin (Mark Rabin), a former programmer who was fired in 2015, recalls how one manager at the general meeting said that Boeing does not need seniors, because their products are already mature enough. “I was shocked that in a room filled with a couple of hundred mostly senior engineers, we are being told in all seriousness that we are not needed …”
A typical jetliner is made up of millions of parts and millions of lines of code, and Boeing has long since outsourced much of the work to suppliers who simply follow detailed blueprints.
Beginning with the 787 Dreamliner launched in 2004, Boeing has sought to maximize profits by providing high-level specifications instead of blueprints and then letting suppliers do the parts themselves. The idea was “they're experts, you know, and they'll take care of these things for us,” says Frank McCormick, a former flight test engineer who later served as a consultant to regulators and manufacturers. "It was just stupid."
Sales are an additional reason for moving work overseas. In return for a $ 11 billion contract signed with Air India in 2005, Boeing has pledged to invest $ 1.7 billion in Indian companies. This, of course, was a boon for HCL, Cyient, and other companies whose programmers were widely used in the computer industry but were not yet involved in aircraft construction.
Rockwell Collins, which manufactures aircraft cockpit electronics, was one of the first aircraft manufacturers to outsource much of their work to India, where HCL began testing their software in 2000. By 2010, HCL employed more than 400 people developing and testing software for Rockwell Collins from offices in Chennai and Bangalore.
In the same year, Boeing, in conjunction with HCL, opened a so-called "center of excellence" in Chennai, stating that the companies would collaborate "to create mission-critical flight test software." In 2011, Boeing added Cyient (then known as Infotech) to its vendor of the year list for design, testing, and software development for the 787 and 747-8 at another facility in Hyderabad.
Boeing's competitors also rely partly on outsourcers. In addition to supporting sales (as mentioned above), aircraft companies claim that distributed design teams are more efficient because they work around the clock. But outsourcing has long been a sore spot for some Boeing engineers, who, in addition to fear of losing their jobs, say it has led to communication problems between teams and errors.
Moscow mistakes
Boeing also expanded its design center in Moscow. In 2008, during a meeting with the chief engineer in charge of Boeing 787, one of the employees complained that he sent the blueprints to the team in Russia 18 times before they realized that smoke detectors had to be connected to the electrical system, Cynthia Cole said (Cynthia Cole is a former Boeing engineer who led the union of engineers from 2006 to 2010.
“Engineering started to become a cheap commodity,” adds Vance Hilderman, co-founder of TekSci, a contractor engineering firm that began losing orders to overseas competitors in the 2000s.
American avionics companies have moved more than 30% of their software development overseas in the past few years, compared with just 10% of European companies, according to Hilderman, a security engineer with 30 years of experience and recent clients that include major Boeing vendors. …
The strong dollar was the key to the attractiveness of this model. Engineers in India were making about $ 5 an hour, now it's $ 9 or $ 10, compared to $ 35-40 for those in the US on an H1B visa, Hilderman adds. But he explains to his clients that the real low cost per hour costs them $ 80, due to the need for control, and says that his firm partially returns customers who need to fix bugs.
HCL, formerly known as Hindustan Computers, was founded in 1976 by billionaire Shiv Nadar and has annual sales of over $ 8.6 billion. According to Vice President Sukamal Banerjee, HCL is a global company with 18,000 employees in the US and 15,000 in Europe, and has vast experience in computing. And I won the order from Boeing for this very reason, and not at all because of the price. He directly states: “We have extensive experience in R&D (Research & Development, research and development work - approx. Transl.)”.
However, while working on the 787, HCL gave Boeing a remarkable price - free, according to Sam Swaro, an assistant vice president who offered HCL services at a San Diego conference hosted by Avionics International magazine in June. He said the company did not take upfront payments for the 787 and began invoicing only on a sales basis a few years later - an “innovative business model” that he proposed to extend to other companies in the industry.
The Boeing 787 model entered service in 2011, three years late, and exceeded budget by billions of dollars, in part due to confusion over the outsourcing strategy. Under the leadership of Dennis Muilenburg, a longtime Boeing engineer who took over as CEO in 2015, the company said it plans to take over much of the work on the latest aircraft.
Engineering Swamp
The Boeing 737 Max became the sales leader shortly after it was announced in 2011. But for ambitious engineers, it was a bit of a swamp, says Peter Lemme, who designed the Boeing 767 autopilot and is now a consultant. The Boeing 737 Max was a 50-year-old design update, and the changes had to be limited enough for Boeing to churn out new aircraft like hotcakes, with minor changes for assembly lines or airlines. “For an engineer, this is not the best job,” added Lemm.
Rockwell Collins, currently a division of United Technologies Corp, won a contract to supply booth displays for the 737 Max and relied on HCL engineers based in India, Iowa and Seattle to do its job. A spokesman for United Technologies declined to comment on the situation.
Contracting engineers from Cyient helped with the flight test equipment. Charles LoveJoy, a former Boeing employee, said US engineers had to recheck plans made in India at night at 7:30 every morning. “We had problems with the Indian team. They fulfilled the requirements, but we could have done better.”
Numerous investigations, including a criminal investigation by the US Department of Justice, are trying to figure out how and when critical decisions were made regarding the 737 Max software. During the crashes of the Lion Air and Ethiopian Airlines planes, which killed 346 people, investigators said the MCAS system pushed the planes into an uncontrolled dive due to poor data from one sensor.
According to Lemma, such a design violates the basic principles of redundancy that have been unshakable for several generations of Boeing engineers. Apparently, no one has ever tested how the software would react in this situation. “It was a deafening failure,” he said. "Not one person, but many people had to think about this problem."
Boeing also said that shortly after delivering the 737-Max in 2017, they discovered that a warning light that could alert the crew of a sensor problem was misconfigured in the flight display software. In a May statement by Boeing explaining why the company did not inform regulators in time, it said the engineers decided it was not a safety issue.
"The general management of the company," the statement says, "was not involved in this audit."
From a translator: After reading the article, I stopped being surprised by the situation in my industry (e-commerce). If there is such a mess with processes in the industrial giants responsible for human lives, then what to talk about in smaller offices. Well, I will add that Bloomberg is of course distorting (there was such an impression), since their task is hype and views, so what is written needs to be divided by two.