RECENT HIGHLIGHTED PUBLICATIONS

Association of Hospital Public Quality Reporting With Electronic Health Record Medication Safety Performance with David W Bates in JAMA Network Open.

Using a regression discontinuity design, we evaluate the causal impact of hospitals receiving negative publicly reported feedback on EHR-based clinical decision support system performance at prevent adverse drug events on performance in the subsequent year. We find that hospitals who receive negative public feedback improve significantly more in the following year, suggesting that public reporting of specific process quality outcomes can be a useful tool for policymakers seeking to incentivize improvement.

Physician EHR Time and Visit Volume Following Adoption of Team-Based Documentation Support with Nate Apathy and Dori Cross in JAMA Internal Medicine.

Using a novel national dataset of EHR metadata and newly developed difference-in-differences event study estimators for staggered treatment adoption, we evaluate the causal impact of physician adoption of team-based documentation support on their productivity and EHR burden. We find that team-based documentation support increased visit volume by 6.0% while reducing EHR documentation time by 9.1%. These effects intensified over time as physicians adapted to the new workflow, and the EHR burden reductions were realized only by physicians with greater team-based documentation intensity.

Assessment of Electronic Health Record Use Between US and Non-US Health Systems with N Lance Downing, David W Bates, Tait Shanafelt, Arnold Milstein, Topher Sharp, David Cutler, Robert Huckman, and Kevin Schulman in JAMA Internal Medicine.

Taking advantage of a novel dataset of EHR audit log measures across the entire Epic Systems customer base, we compare data from 371 health systems in the United States and abroad. We find that compared to their non-US peers, US clinicians spent more time per day actively using the EHR , documented using more automated note text via copy/paste and templates, and received more inbox messages per day. The median US clinician spent as much time actively using the EHR per day as a non-US clinician in the 99th percentile of active EHR use .


See below for a full list of peer-reviewed research publications:

Peer Reviewed journal articles

  1. Holmgren, A. J.,* Apathy, N. C.,* Sinsky, C. A., Adler-Milstein, J., Bates, D. W., & Rotenstein, L. S. (In Press) Assessing Trends in EHR Message Volume and Impact on Physician EHR Time. JAMA Internal Medicine.

  2. Nong, P., Adler-Milstein, J., Apathy, N. C., Holmgren, A. J., & Everson, J. E. (In Press) Current Use and Evaluation of AI and Predictive Models in US Hospitals. Health Affairs.

  3. Strawley, C. E., Adler-Milstein, J., Holmgren, A. J., & Everson, J. E. (In Press) New Indices to Track Interoperability Among U.S. Hospitals. Journal of the American Medical Informatics Association.

  4. Apathy, N. C., Holmgren, A. J., Nong, P., Adler-Milstein, J., & Everson, J. E. (2024) Trending in the Right Direction: Critical Access Hospitals Increased Adoption of Advanced EHR Functions from 2018 to 2023. Journal of the American Medical Informatics Association. ocae267, doi:10.1093/jamia/ocae267.

  5. Sinsky, C. A., Rotenstein, L. S., Holmgren, A. J., & Apathy, N. C. (2024) The Number of Patient Scheduled Hours Resulting in a 40-Hour Work Week by Physician Specialty and Setting: A Cross-Sectional Study using EHR Event Log Data. Journal of the American Medical Informatics Association. ocae266, doi:10.1093/jamia/ocae266

  6. Holmgren, A. J., Adler-Milstein, J & Apathy, N. C. (2024) Electronic Health Record Documentation Burden Crowds Out Health Information Exchange Use. Health Affairs. 024;43(11):1538-1545. doi:10.1377/hlthaff.2024.00398

  7. Apathy, N. C., Holmgren, A. J., & Cross, D. A. (2024) Physician EHR Time and Visit Volume Following Adoption of Team-Based Documentation Support. JAMA Internal Medicine. Published online August 26, 2024. doi:10.1001/jamainternmed.2024.4123

  8. Holmgren, A. J., Sinsky, C., Rotenstein, L. S., & Apathy, N. C. (2024) National Comparison of Ambulatory Physician Electronic Health Record Use Across Specialties. Journal of General Internal Medicine. doi:10.1007/s11606-024-08930-4

  9. Holmgren, A. J., Grouse, C. K., Oates, A., O’Brien, J., & Byron, M. E. (2024) Changes in Secure Messaging Following Implementation of Billing ‘E-Visits’ by Demographic Groups. JAMA Network Open. 2024;7(8):e2427053. doi:10.1001/jamanetworkopen.2024.27053

  10. Holmgren, A. J., Hendrix, N., Maisel, N., Everson, J. E., Bazemore, A., Rotenstein, L. S., Phillips, R. L., & Adler-Milstein, J. (2024) Assessing the Relationship Between Electronic Health Record Usability, Satisfaction, and Burnout for Family Physicians. JAMA Network Open. 2024;7(8):e2426956. doi:10.1001/jamanetworkopen.2024.26956

  11. Apathy, N. C., Biro, J., & Holmgren, A. J. (2024) Consistency is Key: Documentation distribution and efficiency in primary care. Journal of the American Medical Informatics Association. ocae156, doi:10.1093/jamia/ocae156.

  12. Hendrix, N., Maisel, N., Everson, J. E., Patel, V., Bazemore, A., Rotenstein, L. S., Holmgren, A. J., Krist, A. H., Adler-Milstein, J., & Phillips, R. L. (2024) Impact of Response Bias in Three Surveys on Primary Care Providers’ Experiences with Electronic Health Records. Journal of the American Medical Informatics Association. ocae148, doi:10.1093/jamia/ocae148.

  13. Liu, T., Zhu, Z., Holmgren, A. J., & Ellimoottil, C. (2024) National Trends in Billing Patient Portal Messages as E-Visit Services in Traditional Medicare. Health Affairs Scholar. Volume 2, Issue 4, April 2024, qxae040, doi:10.1093/haschl/qxae040.

  14. Apathy, N. C., Patel, V. P., Rolle, T. L., & Holmgren, A. J. (2024) Physicians in ACOs Report Greater Documentation Burden. American Journal of Managed Care. 2024;30:SP452-SP458.

  15. Holmgren, A. J., Oakes, A. H., Miller, A., Adler-Milstein, J., & Mehrotra, A. (2024) National Trends in Billing Secure Messages as E-Visits. JAMA: Journal of the American Medical Association. Published online January 10, 2024. doi:10.1001/jama.2023.26584

  16. Rotenstein, L. S., Holmgren, A. J., Horn, D. M., Lipsitz, S., Phillips, R., Gitomer, R., & Bates, D. W. (2023) System-Level Factors and Time Spent on Electronic Health Records by Primary Care Physicians. JAMA Network Open. 2023;6(11):e2344713. doi:10.1001/jamanetworkopen.2023.44713

  17. Holmgren, A. J., Thombley, R., Sinsky, C., & Adler-Milstein, J. (2023) Changes in Physician Electronic Health Record Use With the Expansion of Telemedicine. JAMA Internal Medicine. Published online October 30, 2023. doi:10.1001/jamainternmed.2023.5738

  18. Pantell, M. S., Holmgren, A. J., Iott, B., Neuhas, J., Adler-Milstein, J., & Gottlieb, L. M. (2023) Social and Medical Care Integration Practices Among Children’s Hospitals. Hospital Pediatrics. October 2023; 13 (10): 886–894. doi:10.1542/hpeds.2023-007246

  19. Sinha, S., Holmgren, A. J., Hong, J. C., & Rotenstein, L. S. (2023) Ctrl-C: A Cross-Sectional Study of the EHR Usage Patterns of US Oncology Clinicians. JNCI Cancer Spectrum. 2023;, pkad066, doi:10.1093/jncics/pkad066.

  20. Rotenstein, L. S.*, & Holmgren, A. J.* (2023) COVID Exacerbated the Gender Disparity in Physician Electronic Inbox Burden. Journal of the American Medical Informatics Association. Published online July 12, 2023:ocad141. doi:10.1093/jamia/ocad141

  21. Cross, D. A., Holmgren, AJ., & Apathy, N. C. (2023) The Role of Organizations in Shaping Physician Use of Electronic Health Records (EHRs). Health Services Research. doi:10.1111/1475-6773.14203

  22. Holmgren, A. J., Esdar, M., Hüsers, J, & Coutinho-Almeida, J. (2023) Health Information Exchange: Understanding the Policy Landscape and Future of Data Interoperability. IMIA Yearbook of Medical Informatics. Published online July 6, 2023. doi:10.1055/s-0043-1768719.

  23. Hendrix, N., Bazemore, A., Holmgren, A. J., Rotenstein, L. S., Eden, A. R., Krist, A. H., Phillips, R. L. (2023) Variation in Family Physicians’ Experiences Across Different Electronic Health Record Platforms: a Descriptive Study. Journal of General Internal Medicine. doi:10.1007/s11606-023-08169-5

  24. Holmgren, A. J., Byron, M. E., Grouse, C. K., & Adler-Milstein, J. (2023) Association Between Billing Patient Portal Messages as E-Visits and Patient Messaging Volume. JAMA: Journal of the American Medical Association. Published online January 06, 2023. doi:10.1001/jama.2022.24710

  25. Grouse, C. K., Waung, M. W., Holmgren, A. J., Mongan, J., Neinstein, A., Josephson, A. S., & Khanna, R. R. (2022) Behavioral “Nudges” in the Electronic Health Record to Reduce Waste and Misuse: Three Interventions. Journal of the American Medical Informatics Association. ocac238, doi:10.1093/jamia/ocac238.

  26. Apathy, N.C., Rotenstein, L. S., Bates, D. W., & Holmgren, A. J. (2022) Documentation Dynamics: EHR note composition, burden, and physician efficiency. Health Services Research. doi:10.1111/1475-6773.14097

  27. Holmgren, A. J., & Apathy, N. C. (2022.) Trends in US Hospital Electronic Health Record Vendor Market Concentration, 2012 – 2021. Journal of General Internal Medicine. doi:10.1007/s11606-022-07917-3.

  28. Tang, M., Holmgren, A. J., McElerath, E. E., Bhatt, A. S., Varshney, A. S., Lee, S. G., Vaduganathan, M, Adler, D. S., & Huckman, R. S. (2022) Investigating the Association Between Telemedicine Use and Timely Follow-up Care after Acute Cardiovascular Hospital Encounters. JACC: Advances. Dec, 1 (5) 1–12. doi:10.1016/j.jacadv.2022.100156

  29. Rotenstein, L. S., Apathy, N. C., Holmgren, A. J., & Bates, D. W. (2022) Physician Note Composition Patterns and Time on the EHR Across Specialty Types: a National, Cross-sectional Study. Journal of General Internal Medicine. doi:10.1007/s11606-022-07834-5

  30. Rotenstein, L. S., Holmgren, A. J., Healey, M. J., Horn, D., Ting, D. Y., Lipsitz, S., Salmasian, H., Gitomer, R., & Bates, D. W. (2022) Association Between Electronic Health Record Time and Quality of Care Metrics in Primary Care. JAMA Network Open. 2022;5(10):e2237086. doi:10.1001/jamanetworkopen.2022.37086

  31. Cross, D. A., Adler-Milstein, J., & Holmgren, A. J. (2022) Management Opportunities and Challenges after Achieving Widespread Health System Digitization. Advances in Health Care Management. 2022:67-87. doi:10.1108/S1474-823120220000021004

  32. Holmgren, A. J., & Apathy, N. C. (2022) Assessing the Impact of Patient Access to Electronic Health Information on Clinician Electronic Health Record Documentation. Journal of the American Medical Informatics Association. ocac120, doi:10.1093/jamia/ocac120.

  33. Kutza, J. O., Hubner, U, Holmgren, A. J., & Esdar, M. (2022) Patient Safety Informatics: Criteria development for assessing the maturity of digital patient safety in hospitals. Studies in Health Technology and Informatics. IOS Press; 2022. doi:10.3233/SHTI220618.

  34. Holmgren, A. J.*, Rotenstein, L. S.*, Downing, N. L., Bates, D. W., & Schulman, K. (2022) Association Between State-level Medical Malpractice and Clinician Electronic Health Record Use. Journal of the American Medical Informatics Association. ocac034, doi:10.1093/jamia/ocac034

  35. Arvisais-Anhalt, S., Lau, M., Lehmann, C. U., Holmgren, A. J., Medford, R. J., Ramirez, C. M., & Chen, C. (2022) The 21st Century Cures Act and Multi-User Electronic Health Record Access. Journal of Medical Internet Research. 2022;24(2):e34085. doi:10.2196/34085

  36. Holmgren, A. J., Everson, J., & Adler-Milstein, J. (2022) Association of Hospital Interoperable Data Sharing and Alternative Payment Model Participation. JAMA Health Forum. 2022;3(2):e215199. doi:10.1001/jamahealthforum.2021.5199

  37. Apathy, N. C., Holmgren, A. J., & Werner, R. M. (2022) Growth in Health Information Exchange with ACO Market Penetration. American Journal of Managed Care. 2022;28(1):e7-e13. doi:10.37765/ajmc.2022.88815

  38. Holmgren, A. J.*, Downing, N. L.*, Tang, M., Sharp, C., Longhurst, C., & Huckman, R. S. (2021) Assessing the Impact of the COVID-19 Pandemic on Clinician Electronic Health Record Use. Journal of the American Medical Informatics Association. ocab268, https://doi.org/10.1093/jamia/ocab268

  39. Holmgren, A. J., & Bates, D. W. (2021) Association of Hospital Public Quality Reporting With Electronic Health Record Medication Safety Performance. JAMA Network Open. 2021;4(9):e2125173. doi:10.1001/jamanetworkopen.2021.25173

  40. Holmgren, A. J., Lindeman, B., & Ford, E. W. (2021) Resident Physician Experience and Duration of Electronic Health Record Use. Applied Clinical Informatics. 12(4):8. doi:10.1055/s-0041-1732403

  41. Holmgren, A. J., Kuznetsova, M, Classen, D. C., & Bates, D. W. (2021) Assessing Hospital Electronic Health Record Vendor Performance Across Publicly Reported Quality Measures. Journal of the American Medical Informatics Association. ocab120, doi.org/10.1093/jamia/ocab120

  42. Holmgren, A. J., Cutler, D., Mehrotra, A. (2021) The Increasing Role of Physicians as Bill Collectors: Destined for Failure. JAMA: Journal of the American Medical Association. Published online July 30, 2021. doi:10.1001/jama.2021.12191

  43. Rotenstein, L. S.*, Holmgren, A. J.*, Downing, N. L., Longhurst, C. A., & Bates, D. W. (2021) Differences in EHR Use Between Adult and Pediatric Primary Care Ambulatory Clinicians. JAMA Network Open. 2021;4(7):e2116375. doi:10.1001/jamanetworkopen.2021.16375

  44. Apathy, N. C., Holmgren, A. J., & Adler-Milstein, J. (2021) A Decade Post-HITECH: Critical Access Hospitals have EHRs but Struggle to Keep Up with Other Advanced Functions. Journal of the American Medical Informatics Association. ocab102, doi.org/10.1093/jamia/ocab102

  45. Holmgren, A. J., Phelan, J., Jha, A. K., & Adler-Milstein, J. (2021) Hospital Organizational Strategies Associated With Advanced EHR Adoption. Health Services Research. 2021;00:1–11. https://doi.org/10.1111/1475-6773.13655

  46. Rotenstein, L.S., Holmgren, A. J., Downing, N. L., & Bates, D. W. (2021) Differences in Total and After-Hours Electronic Health Record Time Across Ambulatory Specialties. JAMA Internal Medicine. Published online March 22, 2021. doi:10.1001/jamainternmed.2021.0256

  47. Co, Z., Holmgren, A. J., Classen, D. C., Newmark, L. P., Seger, D., Cole, J. M., Zimmer, K. P., Pon, B., & Bates, D. W. (2021) The Development and Piloting of the Ambulatory Electronic Health Record (EHR) Evaluation Tool: Lessons Learned. Applied Clinical Informatics. doi:10.1055/s-0041-1722917

  48. Thompson, S. C., Holmgren, A. J., & Ford, E. W. (2021) Information System Use Antecedents of Nursing Employee Turnover in a Hospital Setting. Health Care Management Review.

  49. Holmgren, A. J.,* Downing, N. L.,* Bates, D. W., Shanafelt, T., Milstein, A., Sharp, C., Cutler, D., Huckman, R. S, & Schulman, K. (2020) Assessment of Electronic Health Record Use Between US and Non-US Health Systems. JAMA Internal Medicine. Published online December 14, 2020. doi:10.1001/jamainternmed.2020.7071. Featured Editorial: Sinsky, Christine. (2020) Advancing Practice Science With Electronic Health Record Use Data.

  50. Co, Z., Holmgren, A. J., Classen, D. C., Newmark, L. P., Seger, D., Danforth, M., & Bates, D. W. (2020) The Tradeoffs Between Safety and Alert Fatigue: Data from a National Evaluation of Hospital Medication-related Clinical Decision Support. Journal of the American Medical Informatics Association. ocaa098, https://doi.org/10.1093/jamia/ocaa098

  51. Holmgren, A. J., & Apathy, N. C. (2020) Evaluation of Prescription Drug Monitoring Program Integration with Hospital Electronic Health Records by US County-level Opioid Prescribing Rates. JAMA Network Open. 2020;3(6):e209085. doi:10.1001/jamanetworkopen.2020.9085

  52. Holmgren, A. J., Botelho, A., & Brandt, A. M. (2020) A History of Prescription Drug Monitoring Programs in the United States: Political Appeal and Public Health Efficacy. American Journal of Public Health. no.(): pp. e1-e7. doi:10.2105/AJPH.2020.305696

  53. Classen, D. C., Holmgren, A. J., Co, Z., Newmark, L. P., Seger, D., Danforth, M., & Bates, D. W. (2020) National Trends in the Safety Performance of Electronic Health Record Systems: 2009-2018. JAMA Network Open. 2020;3(5):e205547. doi:10.1001/jamanetworkopen.2020.5547. Featured Editorial: Ratwani, Raj. (2020) Modest Progress on the Path to Electronic Health Record Medication Safety.

  54. Holmgren, A. J., Apathy, N. C., & Adler-Milstein, J. (2020) Barriers to Hospital Public Health Reporting and Implications for the COVID-19 Pandemic. Journal of the American Medical Informatics Association ocaa112, https://doi.org/10.1093/jamia/ocaa112

  55. Levine, D. M., Co, Z., Newmark, L.P., Groisser, A. R., Holmgren, A. J., Haas, J.S., & Bates, D.W. (2020) Design and Testing of a Mobile Health Application Rating Tool. npj Digital Medicine. 3, 74 (2020). https://doi.org/10.1038/s41746-020-0268-9

  56. Apathy, N. C., & Holmgren, A. J. (2020) Opt-in Consent Policies: Potential Barriers to Hospital Health Information Exchange. American Journal of Managed Care, 26(1), e14-e20. Health IT Special Issue.

  57. Holmgren, A. J., & Apathy, N. C. (2019) Hospital Adoption of API-Enabled Patient Data Access. Healthcare. (2019/08/27/ 2019): 100377. doi:10.1016/j.hjdsi.2019.100377

  58. Holmgren, A. J., Co, Z., Newmark, L. P., Danforth, M., Classen, D. C., & Bates, D. W. (2019) Assessing the Safety of Electronic Health Records: A National Longitudinal Study of Medication-Related Decision Support. BMJ Quality & Safety, Published Online First: 18 July 2019. doi: 10.1136/bmjqs-2019-009609. Featured Editorial: Sheikh, Aziz. (2019) Realising the Potential of Health Information Technology to Enhance Medication Safety.

  59. Holmgren, A. J., Adler-Milstein, J., & Chen, L.M. (2018) Participation in a Voluntary Bundled Payment Program by Organizations Providing Care After an Acute Hospitalization. JAMA: Journal of the American Medical Association, 320(4), 402-404. doi:10.1001/jama.2018.8666.

  60. Holmgren, A. J., & Ford, E.W. (2018) Assessing the Impact of Health System Organizational Structure on Hospital Electronic Data Sharing. Journal of the American Medical Informatics Association, 25(9), 1147-1152. doi:10.1093/jamia/ocy084

  61. Holmgren, A. J., Adler-Milstein, J., & McCullough, J. (2018). Are all certified EHRs created equal? Assessing the relationship between EHR vendor and hospital meaningful use performance. Journal of the American Medical Informatics Association, 25(6), 654-660. doi:10.1093/jamia/ocx135. Awarded Editor’s Choice.

  62. Holmgren, A. J., Patel, V., & Adler-Milstein, J. (2017) Progress in Interoperability: Measuring US Hospital Engagement in Sharing Patient Data. Health Affairs, 36(10), 1820-1827. 

  63. Adler-Milstein, J., Holmgren, A. J., Kralovec, P., Worzala, C., Searcy, T., & Patel, V. (2017) Electronic Health Record Adoption in US Hospitals: The Emergence of a Digital “Advanced Use” Divide. Journal of the American Medical Informatics Association. ocx080, doi: 10.1093/jamia/ocx080. Awarded Editor’s Choice.

  64. Holmgren, A. J., & Adler-Milstein, J. (2017) Health Information Exchange in U.S. Hospitals: The Current Landscape and a Path to Improved Information Sharing. Journal of Hospital Medicine, 12(3), 193-198. doi:10.12788/jhm.2704

  65. Holmgren, A. J., Pfeifer, E., Manojlovich, M., & Adler-Milstein, J. (2016). A Novel Survey to Examine the Relationship between Health IT Adoption and Nurse-Physician Communication. Applied Clinical Informatics, 7(4), 1182-1201. doi:10.4338/ACI-2016-08-RA-0145

  66. Holmgren, A. J., Patel, V., Charles, D., & Adler-Milstein, J. (2016). US hospital engagement in core domains of interoperability. American Journal of Managed Care, 22(12), e395-e402. Health IT Special Issue.

*Indicates co-first authorship.

OTHER RESEARCH PUBLICATIONS

  1. Gale, B., Mossburg, S., Holmgren, A. J., & McBride, S. (2023) Technology as a Tool for Improving Patient Safety. AHRQ Patient Safety Network Annual Perspective.

  2. Holmgren, A. J. (2023) The Health Information Technology Special Issue: Tackling New Challenges to Digital Transformation. The American Journal of Managed Care.

  3. Holmgren, A. J. (2019) The Prescribing Panopticon: A History of Prescription Drug Monitoring Programs. The Opioid Epidemic in Historical Perspective: An Anthology.

  4. Holmgren, A. J. & Adler-Milstein, J. (2019) Does Electronic Health Record Consolidation Follow Hospital Consolidation? Health Affairs Blog.

  5. Holmgren, A. J., & Ford, E.W. (2018) Health System Organizational Structure and Hospital Electronic Data Sharing. Proceedings of the Seventy-Eighth Annual Meeting of the Academy of Management.

TEXTBOOK CHAPTERS

  1. Dixon, B.E., Holmgren, A. J., Adler-Milstein, J, & Grannis, S. J. (2022) Health Information Exchange and Interoperability. In: Finnell, J. T.; Dixon, B. E. Editors. Clinical Informatics Study Guide. Springer Nature Switzerland. 2022. doi:10.1007/978-3-030-93765-2_14, ISBN 978-3-030-93764-5