Notes and Translations for the New York Times coverage of healthcare.gov software

by john on October 21, 2013

I fear that the reporters at the New York Times are having a hard time sorting through the language of software contractors in their article today (here) on issues with healthcare.gov. Here are some notes, based on my long experience in “contractor world,” on both sides of the contract:

“Federal contractors have identified most of the main problems”

Translation: Contractors are telling the truth about problems they have known about for some time; and there are still some that they are discovering in the work of subcontractors and poorly-reviewed work.

“some contractors worry that the system may be weeks away from operating smoothly”

Translation: It will be a long time (if ever) before the system is operating “smoothly.” It will probably work, eventually, but there may be workflow issues for consumers because of the complexity of the problem. (Perhaps if the Feds release a good API, an open source front-end may emerge.)

“Administration officials approached the contractors last week to see if they could perform the necessary repairs and reboot the system by Nov. 1. However, that goal struck many contractors as unrealistic”

Translation: It is impossible to fix the system by Nov. 1. Bonus comment: I really can’t stand the use of the word “reboot” in this context since what is being talked about is a computer system – “reboot” works as a metaphor for non-computer systems, but in this context, do we really think “reboot” is the right way to talk? What, we’re going to power-cycle the servers? The authors should probably choose a metaphor from medicine: For instance, they might tell us that after the organ transplants, the Feds will resort to the defibrillator.

“Some specialists working on the project said the online system required such extensive repairs that it might not operate smoothly until after the Dec. 15 deadline for people to sign up for coverage starting in January, although that view is not universally shared.”

Translation: Actually it won’t work “smoothly” before January. The comment that “that view is not universally shared” is a form of schedule chicken — these are other contractors who are trying to make the more realistic contractors look bad by offering overly-optimistic assessments. There are a lot of good articles on schedule chicken you can find via the GOOG, but this one by Peter Schuh is good.

“One specialist said that as many as five million lines of software code may need to be rewritten before the Web site runs properly.”

A note: One specialist picked a number out of a hat. What do we think of that figure, 5,000,000 lines? That’s big, and, if realistic, is including the lines of code for all clients of the main code base. It may also reflect code bloat, etc.

“Senator Marco Rubio, Republican of Florida, said on ‘Fox News Sunday’ that the early failures do not bode well for the rest of the health care law, adding: ‘In the 21st century, setting up a Web site where people can go on and buy something is not that complicated.’”

A note: Health care is one of the most complicated products for a consumer, and the baroque nature of the Affordable Care Act makes it even trickier. As has been noted in several places, many of the state-level marketplaces don’t even have search functions. And why? Because health plans are rarely uniform enough in structure to compare apples-to-apples. Rubio is way out of his depth. What we have here is the case of the petulant manager who berates his staff for product issues when the team wasn’t given enough time and resources to do it right.

“One major problem slowing repairs [...] is that the Centers for Medicare and Medicaid Services [...] is responsible for making sure that the separately designed databases and pieces of software from 55 contractors work together. It is not common for a federal agency to assume that role, and numerous people involved in the project said the agency did not have the expertise to do the job and did not fully understand what it entailed.”

A note: Finally, the heart of the matter. Integration is incredibly difficult, especially with medical data. I would call this “burying the lede.” Want to know more about such difficulties? Try reading a book such as Hacking Healthcare.

“Insurance executives said in interviews that they were frustrated because they did not know the government’s plan or schedule for repairs. Insurers have found that the system provides them with incorrect information about some enrollees, repeatedly enrolls and cancels the enrollments of others, and simply loses the enrollments of still others.”

A note: Now this one is really interesting. Let us recall that insurance companies want to make money, so is it any surprise that they would be grumpy about the accuracy of enrollment data? Maybe they could help out, and provide some open tools for cross-checking and matching enrollee data. That would be hard, by the way, because the insurance companies would be attempting to make matches to data in systems they don’t control.

“Accurate enrollment data is essential.”

A comment: “Essential” for whom? What if the patient’s account of their enrollment was simply accepted as accurate at the point of care?

“Nevertheless, disarray has distinguished the project. In the last 10 months alone, [...] officials modified hardware and software requirements for the exchange seven times. It went live on Oct. 1 before the government and contractors had fully tested the complete system. Delays by the government in issuing specifications for the system reduced the time available for testing.”

A comment: Now this really makes me want to cry, because this is going to be the crux of the matter when the Congressional Committees grill the contractors. It is not going to be pretty. There is going to be a lot of drill-down on matters that the contractors have a hard time explaining, and the committee members are going to have a hard time understanding. It is going to be extremely tedious television, I assure you. I am not surprised that the requirements went through big changes seven times — welcome to the world of software, where, doubtless, the people making the requirements decisions were not experienced in understanding the costs in terms of time and resources. Meanwhile, one wonders what kind of automated integration tests were built when the software was being constructed? I would guess the coverage was weak. Pity.

“The identity management system from Q.S.S.I. [...], was a particular source of trouble when the exchange opened. Change orders show that on Oct. 4 [...] the government asked CGI to help it devise a new identity management system to replace the one provided by Q.S.S.I. But specialists said that approach was abandoned as too risky. Ultimately it was decided to fix the current identity system.”

A comment: Probably a wise decision. This is the kind of project where “Version 2″ thinking could be a disaster.

“According to one specialist, the Web site contains about 500 million lines of software code. By comparison, a large bank’s computer system is typically about one-fifth that size.”

A comment: So now I guess we know why our banking software sucks, too. But 500,000,000 lines? Really? Seriously? Let us remember that in 2007, the Linux kernel has only about 2,000,000 lines, and now has some 15,000,000. Methinks the lady doth protest too much.

  • Pingback: Notes for the New York Times coverage of healthcare.gov software | Rocketboom

  • Pingback: Notes for the New York Times coverage of healthcare.gov software | Enjoying The Moment

  • drewvolpe

    I can understand why, for people who haven’t been involved with building software, it must be hard to understand why a project like this is so hard.

    It seems like a great opportunity for someone to emerge who’s able to explain this with a clear analogy non-programmers can understand. I’m reminded of Feynman dropping a piece of rubber in a glass of cold water to explain what happened in the Space Shuttle Challenger disaster.

  • jdobbinsPHD

    Having spent my career in software development, testing, quality assurance and teaching program management, this is a picture of colossal failure on a scale I have never seen. We got to the moon and back with a system much less than 500,000,000 lines of code. How can anyone trust giving their personal and medical information to such a technological monstrosity? It is begging for identity theft on a grand scale. I hope no one entering their data has a credit card or bank account. If they do, they can kiss their money goodbye. Make sure you have a good lawyer on standby.

  • Pingback: HealthCare.gov – seems to be looking like a “classic” big iron software project | Coldstreams.com

  • Pingback: Healthcare.gov needs a lot more than a reboot | jordan conley

  • red

    The sub-text I pick up in all this is that non-techie bureaucrats can’t possibly supervise or contract for highly technological digital services, just as non-techie members of Congress are inept at superintending or investigating almost anything technological.

  • bigtunatim

    Fifty five subcontractors? This was doomed from the start.

  • Eric

    I think it’s pretty clear that someone pulled the ’500 million lines’ number out of a hat.

  • Mayson

    I use Gmail every day. There’s a simple-to-fix bug that has been there for what seems like monthes, which has changed slightly over its lifetime, that I’ve bitched about numerous times, that’s still there. And Gmail is a lot simpler than the health-care system.

  • gsills

    Funny, I build websites for a living. I’ve been doing it for years, mostly in the health care industry. If you told me that a website rolled out and was not able to handle a million visitors trying to buy health insurance on day one I’d not be even a little bit surprise. I can think of exactly 0 websites that would have worked well under those conditions. Things will be better with time. Situation completely normal.

    A ways back Grady Booch said something that really stuck with me – “All great bit programs started as little programs”. He was right starting big is hard. Programs like Linux are stable mostly because that started small and grew.

  • http://7fff.com jgn

    Personally, I think there can be great communication between non-techie bureaucrats / Congressional representatives, and technologists: But it is hard, and requires a lot of collaboration and teaching back and forth. Both sides have to check their egos at the door and listen, while not being pushovers.

  • Pingback: Must See TV: Congressional Hearings on healthcare.gov issues at Scott Porad

Previous post:

Next post: