Sunday, November 10, 2013

I Crashed the Server: Thoughts on the Debacle

Benefits are complicated.  Really complicated.  I remember at Cigna there was a competition between the payroll people and the benefits people.  Payroll was always on top of its game.  Benefits was constantly floundering.

Payroll is complicated in a large corporation. Taxes in how many states -- we'll leave out international.  And local taxes.  And the deductions that the benefits people feed over.

Payroll was cocky.  Eventually they got their reward.  They got to take over the benefits operations.  And one day I was walking out of a meeting, and a senior payroll guy said to me, "You know, benefits are really complicated."

I think a lot of this complication is unnecessary -- just what you get when you leave insurance companies to their own devices for a century or so.  But it's not going away.

Split Eligibility
Here's an example.  You have a retired couple.  The former employee has just turned 65.  The spouse is 62.  They've both been on the retiree health plan, but now the retired employee is signing up for Medicare.  The spouse is not eligible for Medicare.  What do you do?

The answer is called split eligibility.  The former employee gets Medicare as primary insurance, and the company provides a wrap policy to pick up expenses that Medicare doesn't cover.  Meanwhile, the spouse stays on the old company retiree health policy.

But wait.  It's one policy.  Computer programs are written to provide one set of benefits for one policy.  What do you do?  You write a whole new, much more complicated computer program.

Did I tell you your head was going to hurt?  Mine did, for years.

Benefits are complicated.  And, let's face it, benefits are boring.  So are benefits websites. Who, other than computer geeks, could possibly want to play around in the guts of a benefits website?  Certainly not me.  And yet I went to the meetings for 16 years, my eyes glazing over, my mind wandering to the day I would walk the Elysian Fields.

For this simple reason -- the crashing boredom involved -- management tends to leave benefits websites to the IT people.  Bad mistake.

Remembering Y2K
IT people love it when they're left alone.  Remember Y2K?  Most of the world's computers were supposed to crash when 1999 became 2000, because they could not recognize a year that began in 20 instead of 19.  Big problem.  Created of course by the IT people.  And they solved it.  They said, give us a lot of money and go away for a couple of years, and we'll fix this.  Just don't ask us to do anything else while we're off playing with ourselves, fixing a problem that we created.

And that's pretty much what happened, at least at Cigna.

I Crashed the Server
IT people just aren't good at playing nice with others.  This brings me to the day that I crashed the server.  We'd been working on a new web-based program for some benefit -- I honestly don't remember what -- and we needed to do a launch.  This involved sending an email to our 40,000 employees, asking them to click on a button and go to the website.  Maybe just have a look, maybe do something.  But definitely click the button and go to the website.

Our group -- the communications geeks -- suggested that we stagger the emails to employees over the course of the launch day -- a few thousand here, a few thousand there -- so that we wouldn't overwhelm the system by sending 40,000 people an email at once, having 40,000 people open the email at once, and having 40,000 people click the button at once.  We were a bit worried about whether the website would be able to handle that.

No problem, said our IT confreres.  I believe the word "robust" was used.  As in "robust website configuration," or something.  So we sent out about 40,000 emails at once, and about 40,000 people clicked the button more or less at once, and the server crashed.

Or, more accurately, I crashed the server.  Someone needs to be blamed for such a terrible thing.  After all, the IT people had some cleanup to do, the site wasn't available for awhile, and management was terribly embarrassed that such a thing happened on their watch, while they weren't watching.

The Joy of Lunch
Shortly thereafter, I was at a nice communications meeting at a hotel near the office.  This involved lunch in a large room at tables with tablecloths, and of course an open and forthright dialog among people whose incomes varied by a factor of 100.

At one point the divisional IT vice president stood up at his table down front and said he was very unhappy that I had crashed the server.  I waited a beat, then stood up at my table (which was more toward the back, away from the dais), and said that I proposed to keep crashing the server until we got the systems support that we needed.  For what it's worth, he left the company before I did.

Don't Trust the Consultants
So don't leave these things to the IT people.  Number one.  Number two, don't trust the consultants.  Another story.

We were launching yet another major program -- I don't remember this one either, they all blend together -- and we were at a large meeting in a very nice conference room on a floor to which I had never been admitted before.

We were there with the program's chief consultant, to make sure our communications plans were in order.  I was reliably informed that this fellow drove a Rolls-Royce.  I hadn't expected to speak at the meeting, but Mr. Rolls-Royce, as he went down his checklist, asked a question:  What percentage of employees did we expect to reach with our elaborate program of communications?

Nobody spoke for a while.  Then a helpful vice president suggested that I should answer the question.

"Oh, 80 to 90 percent," I said.  I was mindful of riding in an elevator some time previously.  We'd been encouraging employees to move their 401(k) money out of a fund that we were discontinuing.  The fund had $200 million in it, and for a variety of reasons it was better if individuals moved their money out voluntarily, rather than have us do it for them.

Two men were talking in front of me in the elevator.  One asked the other what he was going to do with his money.  The second man asked the first man what he was talking about.  The first man said, Didn't you get the letter about the fund closing?  (The letter I wrote.)  The second man said, "Oh, I never read my Cigna mail."

As we used to say, again and again, attention to benefits information is very low.  (The low attention levels and the complexity of medical benefits explain why people keep saying on surveys that they don't understand Obamacare.  They never will.)

But back to the meeting with Mr. Rolls-Royce.  He pondered my 80-90 percent answer for a moment, then said he was sure that management would require our communications to reach 100 percent of employees.  And he wrote something on his paper.  I think it was probably 100 percent.

Consultants tell management what they think it wants to hear.  What they say may or may not have a connection with reality.
Some Classic Mistakes
So here's what I think happened with  Lulled by consultants who had no clue of the difficulties involved in benefits websites, management deferred to the IT professionals, who also had no clue of the difficulties involved in benefits websites.

Some other classic mistakes occurred.  Benefits sites always have a hub -- medical, dental, life insurance, 401(k) all need to talk with eligibility and payroll., drawing from a host of government databases and dealing with a host of insurance companies in dozens of states, had that problem on steroids.  This is simply not an easy design problem, and I suspect that it's the basic one.

Bottlenecks.  Okay, who said, Hey, let's get everybody to open an account before they can browse.  Amazon lets you look at the books before you buy them.  Somebody help me out here.  Consumer behavior:  First you browse, then you buy.

Something that I haven't heard about -- and I don't know if it's an issue, but I suspect that it is -- is the ping-pong phenomenon.  At one point at Cigna, we decided to go with an outside vendor for our employees' annual enrollment.  The vendor was in Massachusetts; we were in Philadelphia.  A slender electronic pipe connected us.  Programs that had worked just fine when all the servers were in the same room all of a sudden didn't work so well.

Apparently the programs were used to chit-chatting in a very intense back-and-forth manner.  This round-tripping, as I think it was called, all of a sudden took actual time, and the delays just got worse as the communications pipe got more and more clogged.  Eventually the problem was fixed.  I don't remember how.

Last Story
One last story.  Eventually management decided it was time to do a complete overhaul of its pay and benefits systems.  A lot of the underlying programs we were using were quite antiquated, our hub had been jerry-built over the years with inadequate funding and unrealistic expectations, and frankly I thought a top-to-bottom overhaul was a good idea.  Until I heard about the consultants.

Yet another large meeting.  I managed not to say anything, and transferred to the international division shortly thereafter.  Much later I heard from an IT friend about launch day.  Unbeknownst to the consultants, my friend had stored a backup of the old system on a little-used server (Cigna had a lot of them at that point).   D-Day came.  The consultants triumphantly flipped their switch, turning off the old system and turning on the new one.  According to my friend (this is his story, but I believe him), the new system promptly crashed, and the consultants panicked.  They had no backup plan.  The system they had turned off had been trashed in the process of turning it off.  My friend let them sweat for a few minutes and then he flipped on the backup system that he had stored away.  Nobody on the outside ever noticed that the site had gone down, and a few weeks later the consultants had debugged their system, and the relaunch went seamlessly.

Unfortunately, my friend's option wasn't available to  It wasn't replacing anything.  It was brand-new.  But the problems has been facing are not new, and they're not subtle.

How Do I Feel?
How do I feel about that?  I worked awfully hard to help get the Affordable Care Act passed, and I'm angry that the administration made such obvious mistakes with the implementation.  I feel President Obama let me down on this one.  I'm sorry, but it's personal.

1 comment:

  1. Thank you very much . i think this blog would help me alot…. overthinking is one of my major problems… and i think with this blog i can overcome this problem…