Okay, good evening. I'm Dr. Michael Etzakis. I am the Chair of Registries Committees, and thanks for joining us on a Wednesday night. Where else would you want to be than hearing about why PM&R should really be engaged in big data and real-world evidence generation? We have a great line of speakers talking to us this evening. By way of background, we started this data journey back in 2015. Many experts in our field realized that we needed better evidence to support the things that we do in terms of the outcomes of the interventions that we do for our patients on a day-to-day basis. We all realize the weaknesses of the conventional randomized controlled trial approach to assessing the value and impact of what we do as physiatrists, and so we started a journey on looking at data and data collection and a unified way to measure outcomes on a systematic basis, and this is where we started the registry journey. That was a few years ago. We've come a long way since then. We now have a registry, and we are now embarking on a series of lectures, an educational series, to share the journey with you all, to truly describe some of the things that we've learned and some of the reasons that we feel are compelling that we as physiatrists need to use data collection, either locally or big data collection, to demonstrate the value on a day-to-day basis to improve our quality, improve our outcomes, to measure what we do, but also to measure what we do as specialists and as a field of physiatry in a new age of value-based care. Who's our faculty? We have four great speakers tonight. I'm not going to go into all their backgrounds because I could be here all evening with their incredible backgrounds, but Crystal Price is Director of Informatics at Arbor Metrics. Arbor Metrics is our registry vendor, and they're the vendor for a lot of registries. She has a lot of experience with registries and regulatory advocacy. Michelle Langer did great depth of experience in PROMIS. PROMIS is the patient report outcome tool that we are using in our CADMES registry and is being used in many outcome tools. It's an up-and-coming outcome tool that we'll hear lots about. Then we'll hear from Dr. Catherine McLean and Joel Press, who are at Hospital for Special Surgery. Dr. McLean, Chief Value Officer at HSS, who's led the development of their measurement programs. And of course, Dr. Press, many of you know, past president of our academy and NASS and as a Chief Physiatrist at HSS. In terms of the outline of what we're going to hear from tonight, we'll start to hear about the use of clinical data registries from a big-picture perspective from Crystal Price. Then we're going to move on to learn more about PROMIS and patient report outcomes from Michelle Langer. And then we'll dive into some of the exciting things that are happening at HSS from Dr. McLean and Dr. Press. And then at the end, I'll wrap up with a little bit of an overview of our American Academy of Physical Medicine's active registry. And then we'll move to some question and answer period for about 10 minutes at the end. So I'm going to turn it over now. I'd like to thank AAPMNR for inviting me to speak tonight. My name is Crystal Price, and I'm the Associate Director of Clinical Informatics and Policy at Arbor Metrics. And before we get started, I just want to thank some of my other Arbor Metrics colleagues, Bradley Moore, Kristen McMurphy, and Misty Graham, for their expertise and support in helping me put this together. I'm an employee at Arbor Metrics, and like Dr. Hatzikas said, we are the registry technology partner for AAPMNR. Big data. This is jargon that's been thrown around a lot over the last five to 10 years, but what does it really mean, and how does it pertain to the clinical registry? Patient data can really be classified as big data, especially as societies are the natural conveners for condition-specific registries, with clinical registries being the natural aggregators. However, we shouldn't be lured by the jargon and should be instead focused on our goals of what we want to do with this vast amount of data, and how can we curate this data to best meet our needs? Big data is characterized by the five Vs, volume, velocity, variety, variability, and veracity of data. The volume of data available has really changed. Traditionally, registries looked at single patient encounters and treated them much as an encapsulated point in time, much like the EHRs do. We're in a position where we need to be able to track patients over time to be able to conduct studies using real-world evidence. Volume and velocity are also at play. Data now comes in from many different directions and increasingly more instantaneously. How do we tailor our registries to deal with this influx of data, and how do we appropriately throttle it so that we're getting the right data at the right time for the right purpose? The variety of data is no longer really limited to EHR data, and we're increasingly more focused on other modalities of data, including patient device and also patient-reported outcomes data. Data is extremely variable now, and while standards are beginning to normalize this and make it more regular, it's pretty unpredictable. So lastly, with data coming in from so many different sources and directions and mostly coming in electronically, it's really important that we verify this data to ensure the right reliability and validity, which really coincidentally should be the sixth fee. Registries are the hub for the variety of information that make up big data. Registries really provide the conditions for collection, analysis, and implementation of quality metrics, which in turn sets the stage for creation of clinical standards. Since clinical standards require a blending of large-scale data on patient treatment and outcomes to paint a comprehensive picture of health, it's really imperative to use tools and instruments to achieve this end rather than depending on any single modality like the EHR. So how does all of this relate to real-world data? This is another buzzword in the data science lexicon, but real-world data is basically everything that clinical trials aren't. It's observational data and is representative of how patients really behave in real life. While clinical trials allow us to isolate how drugs or devices work under specific conditions, real-world data really allows us to see how treatments look in large populations over the course of time, which really should be the basis of any modern registry. Real-world data allows us to draw sound conclusions about populations that clinical trials often don't reach, including populations that might be affected like things such as social determinants of health. All of that's well and good, but we haven't really discussed All of that's well and good, but we haven't really discussed how registries collect and use this data. So historically, most registries collected data via case report forms, and this is still kind of the primary way that most registries collect data. Arborometrics also utilizes push-and-pull connections to EHRs and other electronic data systems for automated data exchange, and that is helpful for reducing burden for clinical staff. While patient data is at the core, it's really also imperative that we're collecting things such as administrative data, considering that that provides insights about how healthcare is utilized and gives a more comprehensive picture of a patient's journey through the entire healthcare system. Claims data is really important in filling these gaps as well. Patient-reported outcomes data is really essential in understanding the quality of care. This provides us insights directly from patients and helps us understand what happens outside of clinic visits, procedures, and hospital stays, which is really a crucial component in PM&R data. So we know that setting clinical standards requires actual patient treatment and outcomes from a variety of different sources, and that clinical registries, including patient registries and product registries, provide a rich source of real-world data. In turn, registries become critical tools for generating and using real-world evidence, and that's the clinical evidence that regards the usage, risk, and benefits of a medical product derived from analysis of real-world data. All of this is really necessary to conduct post-market surveillance and research, and it's really the crux of all next-generation registries. In conclusion, societies such as AAPM&R are really heralding next-generation data collection and supporting real-world evidence through leveraging these various modalities of data collection, including patient-reported outcomes, which my colleague will speak about next in greater detail. And up next, we have Dr. Michelle Langer from Northwestern Medicine. Please hold your questions till the end and put them in the chat box, and with that, I'll toss it over to Michelle. Thank you, Crystal. I'll be speaking about patient-reported outcomes today, just to a brief introduction why it is critical to think about this type of data. So, initial definition, a patient-reported outcome is a health outcome directly reported by the patient. So, I want to differentiate that from a health outcome that might be collected by the provider, noted during the clinical interview. It's important to make this distinction because our research has shown that the agreement between patients and clinicians in terms of their symptom reporting is only sort of moderate, and it's not necessarily because, like this cartoon, the doctor is ignoring the patient, but office time is just limited, and patients often can't recall their symptoms verbally on the spot. They will often only share what they think might be relevant, and they won't even mention anything that they might consider socially relevant. They won't mention anything that they might consider socially or culturally sensitive, such as sexual function. So, where does this type of data fit in clinical practice? So, we already have standard data, such as that which is collected in the physical exam, imaging, blood tests, urine tests. How do we collect data that is just as quantifiable in a standardized, reliable way for things like physical symptoms, emotional well-being, function at home, work, school, social health, physical activity, et cetera? So, this is where the patient-prepared outcome measure comes into play, and a measure is just a questionnaire. So, this is how we can quantify how the patient is doing, their level of symptoms or function, and it provides us a standardized, validated way to use this as another piece of relevant data. So, what do you do with this data once you've collected it? So, I'm going to talk through five different ways you might utilize patient-prepared outcomes in the next few slides. So, the first is when the patient-prepared outcome is the primary target of treatment. So, an example is if you are doing a total knee replacement and the goal of treatment is to reduce pain and improve physical function. Another example is if a patient is undergoing cognitive behavioral therapy and the goal is to reduce their depression or anxiety symptoms. A secondary use would be if the patient-prepared outcome is not the primary target of treatment, but it's still an important issue that should be addressed. So, an example here is a cancer patient who might be experiencing fatigue and pain. Their primary goal may be to get rid of the cancer, but secondarily, you want to treat these side effects and other experiences they have that are, you know, negatively affecting their quality of life. Third is when the patient-prepared outcome is a mediator, moderator, or it's some other mechanism that affects the primary target of treatment. An example here is we know that self-efficacy moderates the ability to manage medications and HIV and a variety of other conditions. Depression is also a known mediator for many, many conditions and therapeutic interventions, so it can be quite helpful to measure both self-efficacy and depression. Fourth is when the patient-reported outcome is a variable that's meaningful in your research. An example here is if you want to compare patients who participate in orthopedic follow-up visits and those who don't to determine who might have better physical function after some period of time. And fifth is that patient-prepared outcomes are becoming increasingly important for quality measurement. CMS right now has bundled payment for total knee and hip replacement that incentivizes collection of global health measures. We will be seeing more of these types of incentives by CMS and other payers in the future, more patient-prepared outcome performance measures are developed for quality measurement. And so now I'm going to speak a little bit about PROMIS, the patient-reported outcome measurement information system, and this was a system that started out as an NIH Roadmap initiative in 2004. And the goal is to develop a core set of measures that would be useful for the general population as well as across chronic conditions and that they would be developed using state-of-the-science methodology. And so they could then replace some of these legacy measures that we have that are disease-specific, only useful in certain situations, and it would reduce patient burden because the patient can take a measure at one specialist's office, and it would mean the same thing to another specialist or their primary care doctor. Everyone would speak the same language because it's one set of measures that applies to everyone regardless of their condition, of which patients generally have comorbid conditions, and it can be quite burdensome to fill out specific measures for each particular condition they have. As a result, we now have over 300 PROMIS measures that have been validated across a wide variety of conditions, both for adults and children, and we have translations in Spanish and 42 other languages, and all of the PROMIS scores are T-scores. So there's one metric that once you learn how T-scores work, so it's a norm-based distribution with a mean of 50 and distribution of 10, then you can understand all of the PROMIS measures and interpret them. So these are the core domains for the PROMIS adult measure, just to give you measures, just to give you an idea of the types of domains that are covered by PROMIS. So there's general physical health domains such as fatigue, pain intensity, dyspnea, GI symptoms, mental health, such as anxiety, depression, alcohol use, anger, self-efficacy measures, social health. The main measure is the ability to participate in social roles and activities, but there are also measures for companionship, social isolation, social support, and the pediatric measures are fairly similar as well. So when I spoke about the state-of-the-science methodology that was used to develop PROMIS, one of the key methodologies of computerized adaptive testing. So computerized adaptive testing relies upon item response theory, which is a statistical theory that allows parameters to be estimated for each patient for an outcome question, and so when you have parameters for each question, a parameter that says how related is this question to the underlying construct, the symptom, the level of function that we're trying to measure, and also where does this question measure best along the level of function or symptom that we're trying to measure. So does this question measure high physical function really well? Does this question measure low physical function really well? When you have that type of statistical information, then you can administer CAS, which means every question is tailored based on the response previously. So each patient can get a fairly tailored assessment. So an example is if they get a question for a physical function assessment that asks, can you walk a block? They say never. So then the algorithm is going to adapt and say, okay, this patient likely has a lower level physical function. We're not going to give them a question with a statistical parameter on the upper end. So we're not going to get a question that assesses if they can run a mile because they can't walk a block. And so as a result, the CAS are generally fairly brief with a high level of score precision. And so this helps reduce patient burdens. You don't have to answer so many questions. Helps your ability to do data analysis because you have very precise scores. And so you get better results. And you can cover a wide range of symptoms. And the banks can then spread across that. And the algorithm will pick and choose depending on how the patient is doing. And so if you want to learn more about PROMIS and implementing PROMS, there are a ton of information at www.healthmeasures.net. Both general guidance for implementing PROMS as well as more information about PROMIS if you want to look at different measures, if you want to implement PROMIS in software. Health Measures also covers, it supports three other NIH-developed measurement systems, AFSCME for sickle cell disease, NIH Toolbox, and NeuroQOL for neurological disease. So please check out www.healthmeasures.net. And if you have any questions about implementing PROMIS, also please feel free to reach out to me. And I will now turn this over to Drs. McClain and Pratt. Well, good evening, everyone. It's a pleasure to be here. So I'm going to be tag-teaming this with Joel Press, someone I think that many of you know, and I have a great pleasure to work with him at HSS. So I'm Kathy McClain. I'm the Chief Value Medical Officer at HSS. And I'm going to share with you kind of our PROM journey and how we're going about collecting PROMS organizationally. For those of you who don't know, HSS is a large musculoskeletal health delivery system located on the Upper East Side of Manhattan. We do about 35,000 orthopedic procedures annually and have an outpatient volume of over 300,000 visits a year, so a pretty busy place. We cover the post rectum of musculoskeletal diseases and specialties. And, you know, as folks on this call know, outcomes that are related to pain and functional status matter a lot for musculoskeletal diseases. So we have had at our institution, it's an academic center, and there's a long history of using PROMS initially for research and clinical registries. And as things kind of have evolved, our clinicians, many of whom are researchers, thought, you know, we ought to be able to see those PROMS in when we're seeing patients as well. And so a few years ago, we found ourselves in a situation where we were collecting PROMS in many different ways, lots of different registries, and in a non-uniform way in different clinical settings. And so we kind of took a step back and said, you know, how can we collect PROMS in a way where we can utilize them clinically and also for research, and then also use them for more administrative purposes, such as quality measurements, and in some cases, payment. So this is our concept of PROMS. This is kind of a virtuous cycle of PROMS. On the top half of the schematic is PROM-based health assessment and use of PROMS in the delivery of healthcare for individual patients. On the bottom half is PROM-driven health improvement and the way that we can use PROMS in terms of thinking about quality, for research purposes, and advancing clinical care and science. And so our vision is that, you know, a patient comes in, they see a clinician, we collect clinical data, we collect PROMS, that information all goes into a data warehouse. We happen to use Epic as our EMR. That information that goes into the data warehouse is kind of immediately turned around, and we present PROM normograms and predicted outcomes to our clinicians so they and the patients can look at that information and consider what the different treatment options are and come to a shared decision. So, you know, once the treatment and procedure happens, you know, the patients come back for follow-up or we follow up with them electronically. Again, we collect that information. Ideally, we want to get to the point where we are looking to see, well, did the patient get better the way we expected them to or not? And, you know, we do that now, but I think not in a formal way. And then, you know, that's kind of a virtuous cycle. That information is collected to be used for clinical research, to refine those predictive models for quality improvement activities, public reporting, payment authorization. So we often have to provide PROMS to payers for physical therapy, and increasingly we're seeing requirements to provide a baseline PROM for authorization for some orthopedic procedures as well. And then, you know, with that information, you know, we kind of routinely refine, like, how we're doing in terms of collecting these PROMS. So, like I said, when we kind of started, we kind of had some problems with the PROMS. We had a suboptimal PROM response rate. Our PROM scores and response to patients weren't always visible to clinicians at the point of care. Our patients were sometimes subjected to multiple requests, sometimes for the same PROM on the same day, you know, from the clinicians and the researchers. And then additionally, the PROM data weren't easily accessible for research, right? So they were, you know, a lot of different databases and weren't tied in longitudinally with the information that we have in our electronic health record. So our goal was to develop an enterprise-wide strategy to improve PROM collection rates and data accessibility. The way that we approached this was to put together an advisory committee and working groups. So we've got four working groups, one is operations. So, you know, like, how do you collect the PROMS? Who is in charge of, you know, making sure that the patient, you know, completes the PROM? On the technical piece, what's the platform? How do the kind of electronic pieces work? And how do data from all these different registries get into one place where it can be merged with our clinical data? On the clinical side, and I think this is really important, and Joel might comment more on this, I don't know about you guys, but no one ever taught me about PROMS when I was in medical school and, you know, or residency or fellowship. And so, you know, one of the, while our clinicians are generally clinician researchers and they're somewhat familiar with PROMS, you know, how do you actually use it, you know, in clinical practice? What does, you know, a Koosh junior score of 45 mean or a Koosh junior score of 70? Like, what does that mean? How should you, you know, how does that inform your clinical decisions? I mean, at the same time, you know, patients don't really understand PROMS or the importance of the process. Some education has to go on there as well. And then we have this kind of research group and I'll kind of put a call out, anybody that's interested in collaborating with us, we, you know, we would, we'd love to do that, but we're, our focus in this research group is really about using PROMS for quality measures. So, you know, what is that right threshold? And, you know, how do we develop reliable, valid measures of quality measures based on PROMS? We are all going to be held accountable for them. And we would like to, you know, advance that science and would invite you to, to help us do that, collaborate with us on that. So I'm going to really touch you here today. We have these guiding principles that patients were, and we had all agree with this, you know, across all the different service lines, that patients are asked the right questions at the right time without redundancy or duplication, that the patients understand, you know, why they're completing these. And that's meaningful to the patients that when we collect the data, it's available for clinicians at the point of service. And that we, we get these data into the central data warehouse so that they can be used for research. Our, I'll go through this very briefly. You know, we start off, we have a whole bunch of registries. We have 69. We actually collect over a hundred different PROMS. They were in a whole bunch of different, and still live in different, the research registries in different databases, RedCap, Obert, Ortec, and Viance. And, you know, on the research side, there's a whole bunch of research assistants, you know, running around making sure patients like these PROMs. In, you know, 2016, we determined PROMs to be a standard of care to our institution, and we started to collect PROMIS-10. We determined that we were going to collect a global measure, we selected PROMIS-10, and that we would collect a disease-specific for all conditions. And we got agreement across service lines that, you know, for certain body parts or conditions, you know, clinicians in different areas would collect the same PROM. So, for example, for lumbar pain, everyone's agreed, the physiatrist, the rheumatologist, the physical therapist, the surgeons, that it's the ODI. So that's what we're doing for lumbar pain. Or lumbar problems. So, got to put that into place. We've built 26 different PROMs into our electronic medical record, which is EPIC. We've done data validation, and we have dashboards, which Joel will go through. And we've done a lot of work to improve patient compliance. So, we've changed email languages. I can actually, when we send a note to patients or email to them to collect the PROM, there's a link they can click on. It takes them directly to the PROM. They don't have to kind of go through a bunch of steps in our patient portal. So, with that, I'm going to hand it over to Joel, and he's going to kind of give you the sort of on-the-ground perspective as both the department chair and a clinician. Thank you, Kathy. You know, this has been a great experience for me working with the Value team, because they've been able to help kind of build what we're actually able to look at every day. And, you know, we have 20-some services here, and we're not all as sophisticated as some of the really good services, like our joint replacement people who have been doing this for many years, but we're kind of bringing the physiatry up. So, the next three slides I'm going to show you, there's like three different levels that we look at. We track the scores and the compliance with Tableau. And so, the first thing that you see on the compliance, this is kind of what a service chief would see, or anybody can look at their service and say, this is physiatry. So, you can see our overall completion rate is around 46 percent since we started around 2019-2020. And we can start to look at key metrics by mode with, you know, what about the in-person people versus the telehealth people, and what were the numbers? So, you can see the number of patients that we're looking at. Then down at the bottom, what percent of these have been completed? And it's interesting, you can see, obviously, on the telehealth, we're way above 90 percent on the completions. In the in-person, we've moved from 2020 in the 40 percent. We're getting up to 60 percent. And we're actually now kind of getting everybody kind of bought in, where that number hopefully will skyrocket up as we present this again in another six months or a year. So, this is what it kind of looks like as what's the physiatry service doing, how are the numbers we're getting in terms of completion, whether it's being virtual, new, follow-up, you know, telehealth, or live. Next slide will show us what it would look like for an individual physician. Now, this is from one of the surgical specialties for their joint replacement. And this would be the surgeon's view that he could look at this and say, how are my patients doing with total knee arthroplasties? And he could look at the PROMIS score tens on the number that he has, and wherever it's 500, it's 1,000. And you can see that there's a light blue line, which is the provider, and the dark blue line is the service. So, you can see, how am I doing pre-op, six weeks post-op, six months post-op, one year post-op, in the whole for the patients that I have with this disease? And then over on the right-hand side, they've actually broken it down to what's the MCID for these procedures, and how am I doing as a provider versus the service? So, you start to get a real flavor for how you're doing compared to the service. There may be differences by your, you know, the payer mix and the type of patients that you're seeing, but you're starting to get a sense for it. And as the numbers get bigger and bigger, it has even more meaning to see how you're doing. So, you can kind of do quality control checks even within the departments. And then the next slide, it actually looks at, how does this help me with a patient day-to-day? And I think the thing that's probably helped me the most over the last couple of years, since we've started to institute this, in the patients that are filling it out is, I'm getting a promise, you know, mental health and physical health score at the evaluation and at each recheck. And I can check them, and I do, now we've moved it up, and I've actually moved it up in the epic note where I can see it as almost the first thing I look at is, is their score going up or is it going down? And a lot of times where I'm thinking the patient mentally seems pretty good, and their mental health score on the Promise 10 is really, really low, I start to look at that and say, maybe I need to take a couple extra minutes to figure this out, that there's something else going on here that they're telling me in a Promise 10, but I haven't gotten that information from that patient. So, super helpful on kind of all three levels. I'm looking at the whole service, I'm looking at it as an individual doctor, how am I doing, and then I can look at it patient to patient. So, kind of three different levels there, which are helpful. And then finally, what we're trying to do, the next slide, is we're spending more time educating the patients about this and educating the physicians about it. And I'm probably the least computer savvy person in my entire department, but I do realize that just, you know, a little bit of input to the patient, like why this is important, explain it to it. So, we have a lot of stuff that's information for us on our intranet. We have tip sheets that we give to the patients. We actually have a video now that we have for patients when they make appointments that gets emailed to them, that tells them why the problems are important while they're filling them out. And we're actually going to put one together now with no sound, but subtitles that we'll have in the waiting areas for them too. So, we keep kind of driving home why this is important. And I think what Kathy talked about before, if we can simplify this so that it's like one time to ask this, and again, between me and the spine surgeons or the rheumatologists or the PTs that have all seen them, if we're asking it just one time at each one of the checks and not asking it over and over again and having research assistants call it, it makes it so much easier on the patient and then the information becomes that much better for their clinical care. So, I think with that, we're going to turn it back over to Dr. Hedzikas to take us home. Thank you, Joel. That was fantastic. And it's really, let me get myself started here. It's really clear you guys have done some really groundbreaking work in registries, and it's a clear demonstration of just how powerful registries can be when you have them running and when the whole idea behind data collection is embraced in our organizations. We're very excited to hear about what you've been doing there at HSS. Thank you. Thank you for that. I'm going to take a moment to just talk a bit about what the Academy's registry, where we are. We actually have a running registry, and as of the last month, we are now collecting data from a couple of our sites, and we have a data stream of data. So, our registry is live and running. So, I'm going to take a moment just to talk a little bit about the registry's data and basically what it's collecting, why, and how we're collecting it, just to give a sense for kind of where we are and what we're doing in the Academy's registry. First off, there's a couple of ways of slicing what it is we're collecting. So, I'll try and present in a couple different perspectives just to get a sense for, again, what we're collecting, why we're collecting it, and how we're collecting it. The first slice is where the data is coming from. We have two data sources in our registry. One, it is a connection of some sort to the EMR, and that is moving data from EMR to the registry and resorting it in sensible categories. The second source of data is directly from the patient via a survey, a web-based survey, and that's really called the patient-reported data module, which collects data from patients. So, those are two sources of data. The types of data we're collecting is we want to understand about the patient. So, we have one group of data is patient characteristic demographics, comorbidities, barriers to recovery from our patients. The second group is what are we doing to the patients, interventions. Those are the big grouping of what we're doing as treatment, and the third group is outcomes, and this is where PROM and Likert scale pain scores come in as well. So, those are groupings of data in terms of the why. Speaking about the EMR data, well, this is kind of a long list, but really what you're seeing here is information on the patient characteristics, the treatment characteristics, and that comes from the EHR, and this is obviously in development, but we are collecting data, and we are starting to understand how to regroup into these groups as the data starts coming in from our sites, and this EMR data is in these domains that are coming from the EMR. Again, the data we're getting from low back pain and our stroke groups at this point, it looks very similar with some refinements that are condition-specific, but by and large, the data we are collecting from sites looks very similar between our low back pain and ischemic stroke sites. So, that was the EHR side of data. Now, from the patient data, we are collecting PROMIS is one group of data, and again, we chose PROMIS. It's a condition agnostic tool, so it is going to be, as you saw, we're using PROMIS for stroke and low back pain. We are hoping to use the same outcome tool for other conditions as we expand the registry to other domains. We are looking at performance outcomes like physical functioning and social roles and activities. We're looking at pain level intensity and interference and barriers and comorbidities such as anxiety, depression, fatigue, and sleep. That's PROMIS, and we're also asking additional questions such as return to work, blood thinners, complications, readmissions, and other exclusions at this point such as prior surgeries and workers' compensation litigation claims. So, this is the patient component of our survey. So, that was the what and the why. Now, this is the how. The how, tier one, is our optimal way of communicating with our sites. This is hands-off. Once it's set up, the registry then asks the EMR site for the data, and the data comes back on some regular basis and populates the registry automatically once it's been configured in the front end. Tier two is specific to each EHR, depends on the EHR site, and this requires a little bit more intervention, but it's still mostly automated. And tier three is where the site generates data and then sends it in a fairly flattish kind of file from the site to the registry. Many of sites are starting with tier three and as they get comfortable with the registry process are moving up to the, close to the tier twos and tier ones, which obviously is a more cost-effective way. It's less staff impactful, but does require a little more connection to the EHR. So, that's kind of the quick what, where, and how of the current registry. Like I said, it is running. We have a registry that's operational and we are in our pilot phase, but we are collecting useful and meaningful data at this point. So, any more information you'd like to hear about the registry, please email registry at the academy. Email is up there. Please visit the website or come attend our session at the annual assembly. So, I'm going to stop talking and I'm going to start opening up for questions. This is a good time to raise your hand in the box or post a question in the Q&A. Well, I'm Hilary Stevens, a physiatrist here in California. Beautiful presentations. Thank you so much. And I've been involved with clinical research and recently a VA randomized control trial of care management in Parkinson's disease. A question is about, be it PROMIS or any of these measures, how are you accounting for, A, the environment, sort of the factors in the physical environment? I know this is tough. How do we account for the physical environment? And then how do we account for the care partner? And that will, in all of these measures, any suggestions, any advice? Thanks very much. Do you mean who's filling out the survey? I guess I'm a little clarification of what you're asking. How to decide about interventions and whether they're effective or not. One of the factors is, well, what's in the physical environment? An example, falls in the elderly. If a patient is falling and has a broken hip or has had a stroke, is there any measure of the environment being assessed or what needs to be changed there? And the same in chronic conditions is about the care partner, caregiver burden, those kinds of issues. And I speak from the more complex type of patient in which those issues could play a major role in trying to have an effective intervention. Well, I guess in the iteration of the Academy's registry, we haven't reached that level of resolution of data, but I don't know, maybe Dr. McClain and Dr. Press, maybe you have any comments on how you're operationalizing that to that degree? Yeah. So good question. And I think it's a really important topic and it kind of gets at some of the risk adjustment. And we understand that social determinants impact patient outcomes, including patient report outcomes. And I would say that it's not settled. The science of it is not settled. And it's an area of a lot of research right now. From our standpoint, we do separately than the PROMs collect information on who else is taking care of the patient, caring for the patient, who the care partners are. We understand when we're seeing our patients, what their home situation is. But that being said, I think that we need to do a better job and we're kind of in discussions right now, collecting some information on health literacy and other social determinants to kind of incorporate those. But I would say that we don't, we have to think about them. I don't think that we have the answer and we certainly aren't solid on the risk adjustment piece right now. Thank you. So I have a couple of questions in the boxes. Dr. Luttrell, you have a question about, and I'll paraphrase a little bit. What you're asking is to really get at the true, genuine baseline pain or dysfunction as opposed to how they're feeling at the moment, they're feeling at the survey. And that's a really good question. It's a hard one because you have to be able to say, what did you feel in the last seven days and on average versus right at the moment? And I don't believe we're actually asking questions of that specificity, but it is important because people can be very concrete. I don't have pain right now, but I had terrible pain yesterday and most of the days of the week. So it is an interesting question. I don't know if, Crystal, you have any feedback on that or any of the rest of the group have any feedback on that? I can comment that we actually had a work group that just put together kind of our intake forms for physiatry. And initially what they want to do is just ask, how's your pain today? And then everybody fought very, very hard so that they made sure that they were asking the question, how is it today best, worse than over the last week average for exactly that reason? Because again, everything, a point in time is crazy by one o'clock in the afternoon versus five o'clock in the afternoon is totally different. So you have to have some kind of surrogate for where it is overall. I think that some of it gets washed out or it gets clarified, not washed out, but when you have them repetitively seeing over time, you're going to see, you're going to see it'll regress to the mean a little bit more. And you're going to see kind of what that average really is over time, which is what you're really looking for on these problems, which direction are they going? And that's going to be the most helpful use for information for you. Thanks, Joel. Let's move on to some more questions. There's a question about sharing thoughts about how any registry can be used as a multi-center trial platform. It's an interesting question. Obviously it is one of the potential uses of a registry since it is a national data collection platform that is standardized and cuts down a lot of coordination requirements. But again, that is a sort of a future goal, especially with the Academies Registry. And I don't know, Catherine or Dr. Press, are you at that stage where you're thinking about trials or are you still in QI at this point? I know that for the Academies Registry, we still have a ways to go before we're ready to engage at that level, but it is certainly an aspirational goal of the registry over the next several years. I don't know, either of you want to chime in on that one? So our registries, so we have a lot of very mature registries, and those registries support a whole lot of different research projects. I would say most of those research projects fall into the kind of clinical epidemiology bucket rather than a clinical trial. I will say, though, that as an organization, we participate in INCITE, which is the PCORI-sponsored data registry for New York City. So all of the hospitals in New York City participate in it, and so we have a data feed with specific elements that are just pulled out of our EMR that go into that on a regular basis. And through PCORI, they are doing pragmatic clinical trials using both the INCITE and other similar PCORI-sponsored databases in different cities in the United States. And by the way, those PCORI registries, for instance, the INCITE registry, we contribute to it, but you don't have to be a contributor to access those data. So if a PM&R or any individual investigators are interested in accessing those data, you can just put in a proposal and use those data. There is a question for Drs. Pressman and Klain about how physicians are incentivized to participate in the registry. Is it mandatory? Is it optional? And how are people being encouraged or incentivized to jump on board? I'm going to let Joe give his perspective as the chair. Well, I haven't seen anything changed on my paycheck by doing it, so I don't think there's a financial incentive. I think the biggest incentive has been it helps you take care of the patient, and that's been the biggest sell that we've had in our department is when you start to show people that you can put this up top, you get the number between the different things, you know, there's no carrot, there's no stick. It's just basically this is kind of what the best care is, and this is kind of the expectation. And fortunately, I think most people, it's all an upside. I mean, what we're doing as the physicians is just trying to encourage the patients to do it and tell them why it's important and support the people who are supporting us by actually doing it. So we don't have a incentive. We don't have a stick. Thanks, Joel. I'm going to move over to the chat box now. So there's a question about clinical research, and you say the registry. I'm going to assume that that means the academy's registry. So we are in a pilot phase, which means that we are working through the bugs. We are collecting real data. We're collecting a lot of real data, but we are still trying to perfect our process and look at data integrity and all of the issues of breadth and depth and speed that were discussed before. So it's going to be a few years before we're ready for clinical research. It doesn't mean we can't start thinking about how we might entertain some small multi-center collaborative projects. That certainly is something that we want to do and want to encourage, but I think we're still a couple of years away from that. And when you ask when members have access to the registry, well, certainly participating sites will have access to the registry. It will be, of course, de-identified anonymized data. We are working on data use. We have a data use committee that's going to be emerging for working on data access policies and procedures for accessing data for clinical QI, and that's for the Academy's registry. I'm going to keep moving on. A couple more questions here. There are concerns about IRB. So I think this might be a better question to answer offline. I don't know, Beth, you want to jump in on that? I know you have some experience with talking about talking sites through the IRB issues. Yeah, so I think this question is specifically I'm asking about other registries and other individuals collecting PROMs, if they've had to get IRB approval through their institution to do so. And from our standpoint at the Academy, many times when we talk to institutions, their IRB, their research departments feel like they need to go through IRB approval. We, a PM&RS registry, have received an IRB exemption because this is quality improvement work. This is not research. If a site wants to do research on their own data, just like any research project, you would have to get your own IRB. And most sites that are already collecting PROs didn't need to go through an IRB approval for the PROs they're collecting. So what makes our registry different? That's the million dollar question that we're still trying to answer. So Dr. McLean, Dr. Press, Michelle, I'm not sure if anyone has some thoughts there about the IRB approval process and PROMs. So I can comment on our experience. So for our registries and, you know, all these different registries that we have, we have a system in place whereby there is an IRB that covers certain specified types of research that are going to be for research completed on that registry. And so as long as the research is within the bounds of our IRB approval, we don't have to keep going back to the IRB. However, we do have to have a clinical review committee. So anybody that wants to do a research project, they review the project, they affirm that, yes, this is consistent with our IRB approval, and then those investigators are then, you know, kind of put on the IRB for their specific research project. If it's something that's different, you know, out of scope from what the IRB had previously approved, then you have to, we have to go through and get another IRB. Thank you. That's great. Looks like we are done with the questions and it looks like we are done with time. So I want to thank everybody for signing on tonight. Again, just a reminder, you can reach out to registry at aapmnr.org. You can go to the Academy's website or please attend the seminars in November's Academy meeting. And of course, this lecture is recorded and will be available on the online learning portal of the Academy's website. Thank you very much for attending and we will hopefully see you in our Academy meeting.

data collection

data analysis

physical medicine and rehabilitation

clinical data registries

patient-reported outcome measures

real-world evidence

quality of care

standardized data collection

big data