Wednesday, May 8, 2013

More problems with patient satisfaction surveys



Here are some updates on the patient satisfaction front.

A paper in last month's JAMA Surgery journal noted that patient satisfaction ratings have very little to do with the quality of care provided by a hospital.

The study analyzed data from 31 hospitals that were participated in patient satisfaction surveys, the CMS Surgical Care Improvement Project (SCIP) and employee safety attitudes questionnaires. 

They found that patient satisfaction did not correlate at all with the rates of hospital compliance with SCIP process measures or the opinions of employees about the culture of the institution for half of the categories questioned.

They concluded that "patient satisfaction may provide information about a hospital's ability to provide good service as a part of the patient experience; however, further study is needed before it is applied widely to surgeons as a quality indicator."

What about patient satisfaction and the quality of medical care provided by doctors? 

This is only an anecdote but it does say volumes about the subject.

A New York area cardiologist admitted to defrauding government and private insurers of $19 million. This was described as the largest healthcare scam by a single physician ever recorded in New York or New Jersey. 

Thousands of patients underwent unnecessary and possibly dangerous tests and treatments. He also employed unlicensed and unqualified personnel who treated patients.

As noted by Dan Diamond, managing editor of the Daily Briefing, the Healthgrades patient satisfaction scores for Dr. Katz all ranged from very good to excellent.

In fact, Dr. Katz has received not one, not two, but three Healthgrades Quality Awards, which are still in evidence on their website. I guess $19 million worth of fraud is not enough to impact one's Healthgrades ratings.

Although this next vignette is about customer satisfaction and has nothing to do with patients, it too illustrates the folly of basing one's opinion on satisfaction scores alone.

According to the Consumerist blog, an subsidiary of the magazine Consumer Reports, certain well-known companies have based employee pay raises and promotions on the results of customer satisfaction surveys.

Apparently, the companies considered anything less than a perfect "5" rating as failure. This resulted in employees telling patrons to either give them a "5" rating or if they could not do so, decline to take the survey. 

I have seen this phenomenon in hospitals too. Staff were coached about what to say to patients to help persuade them to give higher scores. 

I think it's called "gaming the system."

For lots more on the subject, type "patient satisfaction" in my blog's search field (upper right corner).

ADDENDUM 5/9/2013

A friend emailed me this comment: "When I take my car to the dealer for service, they tell me they will be sending me a survey in the mail. Then they tell me if I cant give them all '5's, I shouldn’t fill out the survey, instead I should call them and speak to the manager so they can do better next time"


11 comments:

NeuroTrumpet said...

I wonder how many more studies they'll do before concluding that patient satisfaction surveys are worthless and have little or nothing to do with the actual quality of medical care. My guess is they'll keep churning them out until they obtain some semblance of validation. Either way it doesn't matter because these surveys, and the impact they unfortunately have, are not going away.

As an aside, I wonder how many more studies need to show that the new ACGME intern duty hour restrictions create more problems than they solve before they decide to change them again...

Skeptical Scalpel said...

I agree with both of your points. The people who decide these things believe "shoot first and ask questions later."

artiger said...

Did the JAMA study make the front page of the NY Times? Skip that, it was rhetorical.

I'm not against patient satisfaction surveys, but I hope they are carefully weighted in regard to being tied to reimbursement. On the downside, I see over reliance on these surveys leading to overutilization, overprescribing (especially narcs and benzos, but also antibiotics), and increased physician burnout. I guess it could possibly lead to fewer malpractice lawsuits, as we bend over backward to make sure our patients have their every last wish come true. Perhaps it might force us to communicate with patients a little more effectively. I'm trying to be positive here.

I'm sure there will be some coaching going on. After all, when resident surveys began after the work hour limits started, rumor has it that junior residents were "coached" on how to fill those out.

Skeptical Scalpel said...

There is no doubt that patients who don't get pain meds or antibiotics on demand will rate their satisfaction with the doctor lower.

I agree that residents were and probably still are coached regarding such surveys.

CPPA at Vanderbilt said...

Our research at Vanderbilt demonstrates that more can be learned from unsolicited patient complaints than standard patient surveys. Healthcare professionals with high numbers of complaints put themselves and their organization at high risk for malpractice claims, and threaten the safety culture. Using surveillance monitoring, analysis and aggregation of patient complaints, an organization can identify and intervene with the high-risk professionals. Interestingly, high numbers of patient complaints identify the individual,yet the complaints don't have to accurately identify the professional's issue. Being professional means being able to self-regulate. Our soon-to-be-published Vanderbilt study reports that about 70% of intervened-upon professionals are able to correct the situation after being made aware that they stand out from peers.

Anonymous said...

Regarding CPPA at Vanderbilt

What is exactly to stand out from peers? Kinda relativism implicit here? Doctors and waiters: is there a difference? I guess sadly that our originalities (Doctors') vanished in some spreadsheet.

Skeptical Scalpel said...

Thanks for the comments. I agree that there is research showing that docs will modify their behavior somewhat when shown that they are outliers. In this case, they would stand out because they get more complaints.

There is a huge difference between doctors and waiters. Waiters can do as much as possible to satisfy their customers' requests. The more they do, the more satisfied the customer is.

Doctors may have to appropriately say no to patients who request unnecessary drugs like pain meds or antibiotics or unnecessary imaging tests like an MRI. Then you have an unhappy customer who will give the doc a poor rating. The problem is that patient satisfaction clearly has no relationship to quality of care.

Anonymous said...

It's more about economics rather than patient care or patient satisfaction. The scores are going to determine how much health care dollars are going to be paid by the govt.
Soon you will do surgery and if the patient is unhappy for any frivolous reason that may be, you may not get paid.
Dentists make more money than surgeons. However it takes more time to train and technically more challenging to be a surgeon

Skeptical Scalpel said...

Anon, thanks for commenting. You may be right that dissatisfaction may result in no payment. I would not be surprised.

Mark Rudolph, MD said...

I think most everyone agrees that significant criticism can be directed at the Value Based Purchasing program and Patient Experience of Care surveys (HCAHPS). As an internal medicine hospitalist physician who focuses on patient experience issues and training, I didn't find the results of the JAMA Surgery report surprising. Patients don't have an understanding of what comprises SCIP compliance, so I wouldn't necessarily assume that there would be a correlation between their experience ratings and technical protocol compliance. And I hope physicians, students and other healthcare don't take away from these results the incorrect conclusion that pursuing patient experience of care improvement is the wrong thing to do.

We physicians are generally not great communicators. Why? Because it's just plain hard. And most of us have had no training in these matters. Additionally, the traditional methods of rounding and communicating with patients in the inpatient setting are set-ups for patient anxiety, uncertainty and ultimately, errors. Efforts are being made to improve the way we communicate with patients that improve experience of care for patients (and providers) and can only lead to improvements in patient safety and compliance.

We need to work on SCIP compliance AND the patient experience. These are both valid pursuits. The patient experience is not a surrogate for clinical technical competence or performance. And if we fail to value the patient experience as a worthy metric in and of itself, we are limiting our ability to improve the overall care and compassion that is at the core of the profession we signed on to practice.

Skeptical Scalpel said...

Mark, thanks you for your comments. I agree that patients are not going to understand SCIP or what compliance with it means.

Just to complicate things a bit more, there are several papers showing that SCIP compliance does not correlate with wound infection rates. Therefore, SCIP is not a good indicator of quality.

Post a Comment

Note: Only a member of this blog may post a comment.