IRIS Metric Re-Weighting information

What is the IRIS Metric weighting improvement and why is this important?

• In Release 176, ControlCheck will increase IRIS metric weighting for 3 metrics - Med Trends, Variance Trends, and Full Package Waste - to improve identification of risky behaviors that are more suggestive of diversion concern. 

  • This will be applied for both Nursing and OR Care Areas.

  • Hospitals will also need to have iris_weighted_metrics settings turned on via product support. This is already set to ON for most hospitals except for Kaiser - you can check the list of hospitals with the setting OFF here (https://data.bluesight.com/queries/5781)
    
    

• This improvement addresses the following user problems:

  • All IRIS metrics are currently weighed equally, but certain IRIS metrics are more indicative of diversion concern and should have more weight when determining a user’s IRIS score

• Decrease false positive IRIS reviews and aim to highlight more anomalous IRIS users

What are the specific IRIS Metric weight increases and decreases?

• We’ve strategically increased weights: 5x for VT/MT and 2x for full package waste. These are metrics that have been identified in data analysis and customer conversations as the most impactful to review.

• We have also decreased the weight of dispense patterns and mobility to 0.5x. The reasoning is because both of these metrics are more impacted by inadequate source data (e.g. missing administrations, incorrect location mappings), and there is currently no way to exclude event summaries from being included in these metrics. 

• Please note that these adjustments are part of our ongoing efforts to enhance the IRIS scoring algorithm. As we continue to refine and optimize, our focus remains on delivering more accurate diversion detection outcomes for our users. We’ll continuously monitor user feedback and outcomes to ensure these enhancements effectively meet our objectives.
  
  For more details, please see the data/ analysis for driving this improvement.

How do the weight changes affect interpretation of IRIS scores?

• It has always been possible for someone with only yellow metric scores to have a red IRIS score. This is because IRIS is scaled such that roughly the top 1% of providers have red IRIS scores. 

• With the change in weighting, it is more likely for an user with high yellow scores in variance trends, med trends, and full package waste to have a red IRIS. This is intentional to drive review of providers who score higher in the more impactful metrics.

  • In our preliminary analysis, we were seeing a net average of 2-5 cases of providers going from yellow to red scores. This means that we are not expecting a significant increase in review burden compared to the unweighted IRIS metrics. 

Is there any difference between Closed-Reconciled and Closed-Unable to Reconcile variances within the Variance Trends score?

• In Release 178, we are increasing the weight of Closed - Unable to Reconciles to 2x the weight of open and open historical variances. The weight of Closed - Reconciled variances will decrease to half the original weight. Variance Categories excluded from Variance Trends will continue to be supported and not included as part of the Variance Trends and IRIS score.

• Similar to IRIS reweighting overall, hospitals will need to have iris_weighted_metrics settings turned on via product support for this functionality to apply. This is already set to ON for most hospitals except for Kaiser - you can check the list of hospitals with the setting OFF here (https://data.bluesight.com/queries/5781)

Will IRIS metric re-weighting impact historical IRIS scores or only moving forward?

• This improvement will not change any historically saved scores such as flagged IRIS or Monthly IRIS. However, if application users select a historical date range (e.g. on the investigations page or IRIS 1.0), then the reweighting will be applied.

• Any flagged scores or Monthly IRIS scores calculated after Release 176 will leverage this IRIS re-weighting.

What data/ analysis supports this improvement?

In addition to Community and customer feedback on improving IRIS calculation, we analyzed (and will continue to) IRIS Reviews & Investigation data:

• Analyzed 500k current IRIS scores across 844 hospitals to understand how IRIS metrics individually/ collectively drive IRIS scoring
  
  

• Analyzed ~3000 Investigations completed over the past three months to understand which IRIS metrics are associated with investigations closed as confirmed or suspected diversion. 

• Analyzed ~500 IRIS reviews (limited sample) to identify metrics that drive high false positive review outcomes. Specifically, Dispense Patterns will be down-weighted because we found this metric driving more IRIS scores to red and a good amount of those ES are system limitations.