This is not a NASA Website. You might learn something. It's YOUR space agency. Get involved. Take it back. Make it work - for YOU.
Safety

This Is How NASA Covers Up SLS Software Safety Issues (Update)

By Keith Cowing
NASA Watch
December 6, 2017
Filed under
This Is How NASA Covers Up SLS Software Safety Issues (Update)

Keith’s 6 December update: Sources report that the two individuals who spoke with a potential employer of Ben Samouha were NASA employees George Mitchell and Andy Gamble. They were reportedly bragging about how they had done this until they read NASAWatch – and then they shut up.
Keith’s 4 December update: According to a friend who has spoken with Ben Samouha, he has been retaliated against. Two NASA MSFC people became aware he was being interviewed for a new job and called the new employer. Speaking for NASA, they said not to hire him, that he’s trouble, incompetent, makes waves. More to follow.
Keith’s 27 November note: A letter was sent to NASA MSFC management last week by Ben Samouha, a 30+ year veteran in software safety whose career reaches back to the Challenger era. As has been noted previously on NASAWatch there has been a significant amount of internal controversy over safety and software being developed for SLS. Clearly these safety issues remain. People are quitting instead of trying to fight the system, or in some cases, they leave after having been forced out for speaking up about their concerns. As Samouha notes:
“These people have been for a long time (and still are) continuously ignoring or not properly addressing FSW Safety related observations and findings and unethically do not disclose issues to the upper management in order to show a virtual progress in order to keep their jobs. Anyone with years of experience and integrity to Safety can see through these imposters just like I did.”
MSFC To Safety Contractor: Just Ignore Those SLS Software Issues, earlier post
SLS Flight Software Safety Issues Continue at MSFC, earlier post
SLS Flight Software Safety Issues at MSFC (Update), earlier post

NASA Watch founder, Explorers Club Fellow, ex-NASA, Away Teams, Journalist, Space & Astrobiology, Lapsed climber.

59 responses to “This Is How NASA Covers Up SLS Software Safety Issues (Update)”

  1. Bad Horse says:
    0
    0

    If civil servants don’t care if it’s safe, why would they care if it flies?

    • Jeff2Space says:
      0
      0

      I would assert that the engineers and software developers working on the hardware and writing the software most certainly care if it’s safe. Unfortunately things tend to go astray in middle management. They often don’t want to be the bearer of bad news to upper management and often report that progress is going much better than reality. When you have many layers of management, like at NASA, there are many opportunities for the reports to become rosier and rosier as they approach the top of the management structure.

      • Bad Horse says:
        0
        0

        then they should speak up and not let middle mgt do this to the program.

        • Jeff2Space says:
          0
          0

          Hard to do when middle management is the one who’s doing your performance appraisals every year. And the “worker bees” may very well be speaking up, but their communications are likely being “filtered out” by middle management. Attempting to go around middle management can also be a career limiting move.

          I’ve been in this situation before, as a “worker bee” in an unrelated industry. It’s not fun when your boss ignores your input and tells you something like “we have to make the deadline, it’s non-negotiable”. Before that project was even over, I made plans to transfer to another part of the company and burned my bridges when I left.

          When people start retiring, transferring, or leaving for another job at a much higher rate than normal, that should be a sign to upper management that something is terribly wrong.

  2. Vladislaw says:
    0
    0

    Well if the rocket is never really going to be used for humans .. doesn’t really matter if it is safe or not. A couple test launches and it gets canceled.

    • richard_schumacher says:
      0
      0

      Quite so. Only a nutjob would agree to sit atop that overgrown Roman candle.

    • Jeff2Space says:
      0
      0

      When the payload is worth hundreds of millions, or even billions, of dollars, you really want the launch vehicle to successfully place the payload into orbit. The assertion that unmanned launches can be deliberately made “less safe” than manned launches is largely a fallacy due to the high cost of payloads.

      The exception, of course is escape systems. Manned vehicles typically want an escape system just in case the launch goes awry so that astronauts don’t get killed. Unmanned vehicles typically forgo escape systems knowing that a launch failure means complete loss of payload.

    • Bill Housley says:
      0
      0

      Your point is well taken.

      The program is troubled enough. If the un-crewed test flight craters, then the program is done…so suppression of safety concerns serves no purpose whatsoever. It’s stupid on all fronts.

  3. numbers_guy101 says:
    0
    0

    Ummm…nice…someone with a backbone here. A rare breed, hardly observed anymore in the wild. More to come…

  4. TJ McDermott says:
    0
    0

    It sounds like there is a serious problem, but Mr. Samouha’s message is gravely weakened by his repetitive name-calling, poor grammar, and general rambling.

    Is Mr. Samouha planning to file a whistleblower lawsuit?

    • chuckc192000 says:
      0
      0

      I agree — if his writing is of such poor quality can you really believe his complaints about software quality? He sounds like a hothead who was very difficult to work with.

    • Sam S says:
      0
      0

      Yeah, I think the open letter route is a good path if you have legitimate concerns, which Mr. Samouha clearly has, but you absolutely must “stick to the facts” and avoid sounding like a 3 AM tweet from Donald Trump.

      For example, Mr. Samouha’s argument would be much better received if he completely removed the vitriol from paragraphs such as this:

      “If you look at the bottom of this form it clearly has space to include such terms. However an incompetent and very rude and ignorant people such as George Mitchell forces you not to do so.”

      And reworded them such as this:

      “f you look at the bottom of this form it clearly has space to include such terms. However, George Mitchell does not allow use of this space for its intended purpose.”

      It still names names, which I am completely onboard with when dealing with safety issues – accountability must be enforced in such situations – but it makes it obvious that your problem is not with George Mitchell the person, but with George Mitchell the technical leader.

      As it stands, I honestly don’t know if Mr. Mitchell is the devil Mr. Samouha believes him to be, or if Mr. Samouha is just a very emotional/stressed person that is suffering from a persecution syndrome. At this point Mr. Samouha is doing as much damage to his own cause as he believes Mr. Mitchell is.

      I have experience writing embedded software for functional safety in industrial automation, so I understand the technical arguments about proper testing and hazard analysis Mr. Samouha presents. But I also understand that even in safety-critical systems, there is enough and there is too much, and even objective requirements are subject to interpretation and review, so long as all knowledgeable voices are heard and respected.

  5. NArmstrong says:
    0
    0

    Key terminology might just be “virtual progress”. Its very easy to talk about the great progress being made, but when, year after year, there is nothing to see, nothing to show for the effort, nothing to show for the tens of billions $ spent, you have to wonder what is going on. When you compare the money and the time spent with prior programs, why the lack of visual evidence of progress?

  6. Bad Horse says:
    0
    0

    Only a fool would trust the NASA IG @ MSFC. They are corrupt. Go to the media or congress.

  7. TL says:
    0
    0

    Well the program was publicly cancelled, then quietly nothing happened.

  8. Keith Vauquelin says:
    0
    0

    If true, it sounds like the lessons of the past have already been subordinated to politics. Keep the heat turned up, Keith.

  9. Christopher John Miller says:
    0
    0

    Not surprising! Just another example of NASA sweeping risk under the carpet. A previous Pressure Systems Manager (PSM) at Kennedy Space Center recommended a hydraulic system be shut down (the system was not certified per Agency and KSC Safety policy) until, as a minimum, a hazard analysis and risk assessment was developed and system risk accepted by the appropriate Center risk review panel/board. When KSC Safety & MIssion Assurance (SMA) management learned of the PSM’s recommendation, the PSM was told by the KSC SMA Director “we are not here to stop work!” and the employee was removed from the PSM position within a week due to “poor performance” (the SMA Director later stated in an affidavit “I could have put the PSM on a Performance Improvement Plan and then fired him”). While assigned to KSC Ground Systems Development & Operations (GSDO) SMA Division, an experienced Sr. Systems Safety Engineer (GS-14) was tasked with developing hazards analyses for “legacy” shuttle equipment being repurposed for the SLS program. KSC SMA GSDO managment did not “appreciate” hazards identified by the Sr. Engineer and informed the engineer “THE’ Program has accepted the legacy shuttle equipment “as-is”; the engineer stated the Program deserves to know what risk they are accepting “as-is”; Sr. Engineer’s task was reassigend to a GS-7 who was directed by SMA GSDO to put “NA” for identified hazards (a portable pressure system using relief valves that are not permitted to be used at the Center). It is more than just software

    • ThomasLMatula says:
      0
      0

      Yes, it seems they have forgotten the importance of preventing accidents by encouraging employees to seek out problems and fix them.

  10. Bill Housley says:
    0
    0

    Sniff, sniff…I smell a lawsuit.

  11. ThomasLMatula says:
    0
    0

    So NASA wants CCP to have a LOC of only 1 in 270 flights, but is unwilling to hold the SLS/Orion to the same standard. Why am I not surprised?

    BTW does anyone have the current risk estimates for the Soyuz the astronauts are flying on? In expect its less than the CCP NASA requires.

    • Michael Spencer says:
      0
      0

      Those LOC calculations always seemed reliably fishy to me, based as they are on far to many assumptions to support any significant figures at all.

      • Donald Barker says:
        0
        0

        How about real numbers then. Soyuz, 39 manned flights since 2000 to ISS, no LOC. New take that back to the 1960s.

        • ThomasLMatula says:
          0
          0

          Smithsonian Air and Space had an article on Soyuz safety in 2010. Little seems to have changed about it.

          https://www.airspacemag.com

          Spaceflight Safety: Shuttle vs. Soyuz vs. Falcon 9

          By Mike Klesius
          airspacemag.com
          March 31, 2010

          The numbers as of 2010…

          “Soyuz has orbited 250 people, not including
          two successful aborts: Soyuz 18a in April 1975, which occurred late in a launch 90 miles high, and Soyuz T-10-1 in September 1983, on the launch pad. The program has suffered four fatalities: one on Soyuz 1 in April 1967, and the other three on Soyuz 11 in June 1971. That’s one Soyuz fatality for every 63 people delivered to orbit. Based on those ratios, Soyuz is a little safer.”

          So you may look at it as they have solved the safety issues with Soyuz, or Soyuz is just that much closer to its next accident.

        • Daniel Woodard says:
          0
          0

          The historical record on Soyuz is good, but both the spacecraft and the launch vehicle had some recent anomalies that suggest problems with quality control. https://arstechnica.com/sci… The core problem is with the requirement for an extremely precise statement of risk of loss of crew for a vehicle that has not yet flown. Such an estimate can only be generated by analysis of anticipated failures. Most real failures are unanticipated and therefore not included in the analysis.

          • fcrary says:
            0
            0

            One problem with the extreme precision is that it may drive costs without any real benefit (or, possibly, cause harm.) How much does it cost to reduce known risks of crew loss to one in 270 (0.37%)? If the unknown risks are closer to 2%, is that really worthwhile? Or, could the money be better spent making the system robust to unknown failure modes? At least, in the case of Falcon9/Dragon 2, the escape system seems robust. It would probably handle a fair number of unknown failure modes.

        • Michael Spencer says:
          0
          0

          Good point. I should have been clearer; I was thinking about calculations for vehicles not flown and the fact that failures are most often unanticipated and therefore not included in these ‘calculations’ (see Dr. Woodard’s comment, below).

          And actually I had in the back of my mind the scary headlines about Cassini’s RTG, although there are plenty of other examples. Anybody remember that? The odds of plutonium entering the food chain?

          I was thinking at the time that an awful lot of assumption went into those ‘calculations’. And of course the inimitable Michio Kaku gets in on the action.

          https://www.csmonitor.com/1
          ————————
          I had a marvelous high school chemistry teacher who taught me (at least) two things that have helped me understand science:

          1. Observation: he challenged a bunch of high school juniors to list everything they could while observing a burning candle. I got to a dozen or so, as I recall, then gave up. His list was more than thirty unique things he noticed while looking at a burning candle.

          2. And, to the point here, sigfigs, or significant figures. Decimal places show up in the oddest spots, and often by people who should know better.

  12. Jeff2Space says:
    0
    0

    Wow, that’s really messed up. I hope he finds a new job that has a far less toxic working environment. Engineers and software developers need to be able to be open and honest with management even when they’re delivering bad news. Shooting the messenger is never the right way to fix a problem. Don’t they teach that in management school?

    • ThomasLMatula says:
      0
      0

      You are assuming that those managers have Management degrees. I expect that as with most of NASA they have degrees from STEM fields and learned their management on the job, in training sessions or in “Executive” MBAs. As such they are probably putting their technical opinions against his not recognizing the bias they have.

      • Donald Barker says:
        0
        0

        Having a Management degree of any kind does not mean you understand your or any other humans behaviors or psychological responses.

        • ThomasLMatula says:
          0
          0

          No, but you know how to put processes in place that encourage honest feedback and maximize the input of employees into decision making. That appears to be what is missing here.

          • fcrary says:
            0
            0

            I’m afraid the degree simply means they have paid tuition, and taken and passed courses on the subject. How many people simply see an MBA as a necessary piece of paper for a career in management, and go through the motions? How often do people get the degree and then go on to manage in ways which are contrary to the business school’s curriculum? (Before you say they wouldn’t last long or be successful if they did, remember than the success criteria for NASA and aerospace contractor management is a bit different from what you might be used to.)

          • ThomasLMatula says:
            0
            0

            Yes, unfortunately that is the problem with students with STEM degrees that earn an MBA. They already learned a lot of bad management practices before getting the degree and its difficult to unlearn them. But because they think they understand management they treat it just like a check box and only do the minimum work needed to pass instead of opening their minds to real learning available. That is why in my classes I always teach them some of the history behind the ideas in an effort to open up their minds.

            Most STEM folks are looking for basic laws when in management you are dealing with too many variables to form anything more then generalities. As a result they have a hard time understanding that good leadership and good management are as much an art form as a structured field of study.

            Imagine for example trying to do chemistry if Iron had the ability to change its characteristics at a whim as employees change their motivations and viewpoints. The nice thing about most science is once you have the equation it is just a matter of plugging numbers in. F=ma always works the same as long as units are consistent and you measure them correctly. That is why I always enjoyed my science and engineering classes, because they were so easy.

          • fcrary says:
            0
            0

            I wouldn’t say this problem is unique to people with a STEM background. But you do make a good case for them to be more prone to it.

            One of the other problem I’ve noticed is managers who confuse good management practices with management practices which make the manager’s job easier. Things like, when someone complains about a problem, assigning the complainer with the job of fixing it. There are good sides of that, but it’s also de facto punishment for complaining (and discouraging complaints does make a manager’s job easier.) I suspect this may be more common from people who learned to manage on the job; it’s easy to pick up bad habits from the example of more senior managers.

  13. NArmstrong says:
    0
    0

    The only way to fix problems like this is to publicly identify the culprits in management. An appropriate review mechanism should assess whether the individuals in management have undermined program goals, endangered health or safety, and in some cases broken the law, and if so, they should be removed from management. I know of one NASA Center where managers like this have been promoted into Director’s positions instead of having been tried, found guilty and removed.

  14. Michael Spencer says:
    0
    0

    If the guy is seen as a PITA at NASA, why not help him leave? What’s the motivation for blackballing him?

  15. tutiger87 says:
    0
    0

    I hated going through CMMI training. But, as my career went on, I learned to appreciate it and clearly saw the value of it. But any process put in place is only as good as the humans who are supposed to follow it.

  16. Bad Horse says:
    0
    0

    MSFC is rotting out because of civil servants like the ones referenced in the letter. A national treasure is being destroyed because of corrupt, incompetent government leadership. A sick culture needs to be fixed.

    • Michael Spencer says:
      0
      0

      ‘government’ leadership?

      Without picking a fight— it’s just dispassionate curiosity, from an outsider — why include the word ‘government’? Isn’t it the case that the glorious history of MSFC has been as a government agency of some sort?

  17. numbers_guy101 says:
    0
    0

    This sounds like something the NASA IG should be asked to investigate by Mr. Samouha. A key point will be if the NASA personnel called the new company potentially hiring Samouha proactively, vs. stating a recommendation if called. Was the call as a reference too, as perhaps the hiring company violated policy as well by calling NASA personnel.

    One of the first things I learned as a newbie a very long time ago was that formally NASA and contractors hiring are seperated by barbed wire fences, the only exception being when there is a particular person with specific credentials intrinsic in a bid. Even then there being systems in place as to what info is valid to consider.

    • Daniel Woodard says:
      0
      0

      Just curious if you have ever asked the IG to investigate something. Other than the anonymous “hotline”, is there a formal procedure? Just wondering if thier investigation targets are internally determined or actually responses to external concerns.

      • numbers_guy101 says:
        0
        0

        While I have supported the IG over the years, just as any employee is told they are formally expected to (i.e., reminder emails about such) if requested, providing any information and holding nothing back, I can’t say I understand the process by which anything is investigated, or not. My intuition tells me a mix of policy (any major programs) and large doses of judgement and politics enter into the mix.

        Remember CAIB, afterwhich O’Keefe said “if people don’t want to use OIG hotline, or the NASA Safety Reporting System, there are any number of ways to raise issues. We want to create yet another system such that anyone anywhere at anytime of day who feels that they cannot raise an issue or report a problem can do so. We will be creating an ombudsman functionality at NASA – one seen in other organizations – such that these issues are run to ground.””

        http://www.spaceref.com/new

        Also, regarding private sector employees, applicability just as with NASA –

        http://spaceref.com/news/vi

        • Daniel Woodard says:
          0
          0

          TMK the only “official” path for reporting to the IG is via the “anonymous hotline”, which is just an answering machine rather than a real person. There is no way to tell if any such “public” reports have resulted in an actual investigation.

    • Michael Spencer says:
      0
      0

      There’s a line between gossip and news. My own view is that this report is thin, too thin, adding very little gloss to the fine reputation of this website.

      One would want to know more about the specific complaints made by Dr. Samouha. I’d want to know more about this complainant, too, in order to establish bona fides, so to speak (as if I’d be qualified). The nature of the alleged calls made by the two named people needs more detail.

      These are serious charges, after all. The actions of the two named people, if true, are appalling.

      Nonetheless, there are so many accusations flying around these days with the assumption that accusation = guilt that sometimes the treatment of those accused seems unfair and unAmerican.

  18. Tally-ho says:
    0
    0

    Talk about illegal. Any way to rat out those MSFC people? He was essentially getting out of the way, but they had to get their pound of flesh. When I started at NASA, my manager told me, you have to understand that working here is like being in high school again. It’s the same level of politics. I’d never seen anything like it in the private sector. You can deal with competent assholes, but there are so many incompetent assholes that wouldn’t last 2 minutes in the private sector.

    • chuckc192000 says:
      0
      0

      It’s SO illegal that I find it hard to believe they would do something like that. Samoutha’s original letter was very unprofessional — I find it hard to give him a lot of credibility. I think he mentioned he was around during the Apollo era (or early shuttle days). Is it possible he just doesn’t comprehend modern software engineering techniques? He should have cited some examples of “bad things” he observed in his letter.

      • fcrary says:
        0
        0

        Unfortunately, I’ve heard threats of similar things, so I have less trouble believing this.

        • Daniel Woodard says:
          0
          0

          There are good managers who can accept criticism, but no one blows the whistle on them because it isn’t necessary; they will listen to you. If you attack a powerful man who won’t accept criticism, you are likely to be crushed.

      • Michael Spencer says:
        0
        0

        Indeed these is very little detail in these reports.

  19. Matthew Black says:
    0
    0

    I once worked in the 1990s under a Sales Manager and director who never failed to remind people that he had a ‘Degree in Business Administration’ and knew EVERYONES’ job better than they did. Micro-management, public criticism, constant promises of bonuses (never fulfilled), gaslighting, undermining of good ideas ‘not invented here’ and whisper and smear campaigns against anyone who crossed him for the slightest reason or disagreement. He held meetings to ‘hear better ideas than mine’ – then proceeded to eye-roll, smirk, yawn and constantly interrupt anyone with ideas or procedures alternate to his own.

    The last straw was when he made an error that potentially lost the company hundreds of thousands of dollars and this was exposed during a staff meeting by my colleague, taking calm and sweet revenge. The manager with the ‘Degree in Business Administration’ huffed and bullied and interrupted constantly during this disclosure – saying that ‘this or that’ didn’t happen. Then proof was presented seconds later that made him purple with rage and embarrassment. He finally shut up. The mistake was corrected, and the potentially lost customer was placated. The loss ended up being about $3,000 not $300 thousand.

    A couple days later, the person who had blown the whistle had their car badly vandalized. Other incidents happened. The whistle blower left, but found it difficult to get other jobs as the ‘Degree in Business Administration’ guy somehow found out about most of the whistle blower’s applications – and proceeded to bad mouth them from one end of our small country to the other. Does this TRUE story resemble what has happened to ANYONE reading this?! 🙁

    • mfwright says:
      0
      0

      “The whistle blower left, but found it difficult to get other jobs”

      I never could understand how such ‘Degree in Business Administration’ guys continually able to cause damage. Recreating the capability to put a man where Gagarin went to more than 50 years ago seems like something in the good-luck-with-that dept.

  20. Paul451 says:
    0
    0

    On the top right of your comment, there’s a down-arrow. Click that and one of the options is “Delete”. Click that and your comment goes away. Usually.

    [As a belt’n’braces, because… Disqus… I usually hit Edit and replace the contents with a null or period first, then Save then Delete.]

  21. NJK1024 says:
    0
    0

    Finally had time to read the full missive. I certainly hope his concerns are addressed especially considering human life is involved, but, this letter, oh my.

  22. Bill Housley says:
    0
    0

    Ya. Oops. Keith Cowing, kicking butt and taking names since…how long have you been doing this?

  23. Keith Vauquelin says:
    0
    0

    The details Keith shares represents another gross abuse of power – perhaps the perpetrators are not paying attention to the tsunami-like cultural shift taking place in our country. Using power to marginalize or destroy another person will never cease – but, people are far less tolerant of such egregious behavior and will now publicly speak up.

    When the US, at minimum, loses the investment on SLS through the loss of a vehicle and payload, or at maximum, potentially kills another crew, the NASA employees who are involved with downplaying software and safety issues, if not dealt with now, swiftly and effectively, will be nowhere to be found.

    Our taxpayers will loses their investment in the nation’s future, and the families involved will endure hardship and struggles which can be avoided.

    Keep the heat on HIGH, Keith.

  24. Tally-ho says:
    0
    0

    Wow, just read his letter. He really burned the house down. I don’t suppose it’s possible that some of these safety issues could be listed there w/o ITAR issues, maybe in a generic way.

  25. Tea Monster says:
    0
    0

    Everyone is talking about the legality of this, which is valid. If certain managers have gone on record that they have followed these practices, then an interesting libel case may be constructed.

    While not to minimise Mr. Samouha’s plight, I am more concerned that eventually, 3 astronauts will be sitting in a capsule on top of this thing and attempt to launch into space. Has nobody learned from what happened with Apollo 1? Do we really want a repeat of that?