BC’s Dr. Matthias von Davier on TIMSS & K-12 Global STEM

The Learning Curve – Dr. Matthias von Davier 1/31/2024

[00:00:00] Alisha: Welcome back to The Learning Curve podcast. I am your cohost today, Alisha Thomas Searcy, and I am joined with my new friend, Charlie Chieppo. How are you?

[00:00:34] Charlie: Good, Alisha. I’m happier that I’m here with you, and I’m looking forward to this.

[00:00:39] Alisha: Same here. It’s going to be a very interesting conversation today. So of course, before we get into that, it’s time for our stories of the week, and Charlie, I’ll go first.

Charlie: Go for it.

[00:00:47] Alisha: I really enjoyed reading this interview on Education Next between Frederick Hess and Nina Rees. As we know, Nina Reese is the outgoing president of the National Alliance for Public Charter Schools —

[00:01:05] Charlie: And a recent guest on The Learning Curve.

[00:01:06] Alisha: Exactly, was our guest recently, and so I’m liking these interviews that she’s doing, and we’re learning a lot as a former legislator, you know, I did a lot of work in the charter school space, and so, I learned a lot from this, interview.

[00:01:19] Alisha: And so, a couple of important things, I think, to point out: Number one, she was very frank about kind of the politics of charter schools, where we are. One of the most important things I think she pointed out was all the research that’s been done with CREDO. That while charter schools are certainly outperforming their traditional public schools where they exist as a whole, they’re not necessarily outperforming traditional public schools across the country.

[00:01:50] Alisha: And I think it’s important to say that. One of the things that’s interesting to me in this charter school movement and conversation is that I think some people expected charter schools to come in and like, be all things to all people and knock it out of the box when it comes to student achievement in every area.

[00:02:07] Alisha: And certainly, we want that to be the goal. But she also points out something I think very, very important, which is this marriage that she calls between the charter school movement and the accountability movement has really stifled innovation in the charter school space. And if we think back to the ‘90s, when this conversation started, and we wanted to see more innovation in the traditional public school space, I think we thought charters were going to be the entity that helped to do that.

[00:02:39] Alisha: But her point is, we’ve been so focused on student achievement and yes, taking on students just like the traditional public school system, where they are behind in many ways, that charters have not been able to focus on innovation. And so, I think this interview was so honest. I thought it was spectacular. And the title of it is very good, right? Honestly Assess Your Strengths and Limitations. And so, I think any good leader, whether you’re leaving the job or just getting into whatever, you’re in the middle of the job, you have to take a moment to honestly assess your strengths and your limitations. And so, the final thing that she talks about is, you know, what’s the advice that she’d give to those who are still in the charter space still doing this work? And she said something that we all know that we should be doing, but I’m not sure we do it well, which is coalition building — to find partners who agree with us, who understand the need to have public school choice, no matter what it looks like.

[00:03:36] Alisha: If it’s a charter, if it — and these are my words — but charter magnet, traditional, transferring within the district, whatever it is, that you need coalitions and people who understand the values of the charter school space and leveling the playing field and what it is that we’re trying to do. And I’ve always believed that no matter what your issue is, if you’re a one-issue person for me, I’m education, but it’s also important to me to show up with other coalitions for things that I believe in, like voting rights, as an example. And so, this notion that we have to do a better job of coalition building is so important, so powerful, great article. I think Nina’s work is going to be missed and I hope that she’ll stay involved in the movement and keep having these really important conversations as we move forward in the charter school space.

[00:04:24] Charlie: Well, you know, Alisha, the thing that you bring up that really resonates with me as someone who’s been involved with charters for a long time is Nina’s comment about how charters have for a variety of reasons not always encouraged innovation, because I think, in Massachusetts, that has certainly always been the case.

[00:04:42] Charlie: And at least here, it’s been two things. One has been exactly what you say you know, obviously, continuing to have strong academic outcomes is often politically an existential issue for these schools.

Alisha: And the basis for which they get renewed, right?.

[00:04:57] Charlie: Exactly. So, they’re not encouraged to take chances to take risks. You know, the other thing is, at least here, and it may not be this way everywhere, but boy they were quickly almost pigeonholed into this. by an ever sort of increasing set of regulations around them as they became more and more controversial. But it’s a very interesting point and certainly one that resonates with me as I think about charters.

[00:05:22] Alisha: Absolutely. But I do hope that as we move forward, that there is more conversation about innovation. And we think about those regulations that we put in place because we need it. In public education, we do.

[00:05:33] Charlie: And, you know, here in Massachusetts. In fact, it became part of the law that the law became that essentially to have a charter school above what was a much older cap on the number of them. It had to be a school run by a proven provider, which itself just puts a lid on innovation.

Alisha: So exactly.

[00:05:52] Charlie: But you’re right. Very interesting stuff. Very interesting stuff. So exactly.

Alisha: So, what do you have?

Charlie: Well, I’m going to talk about piece on PBS about the shortage of special education teachers. And so, I have to admit, right up front. I’m the parent of two special education students. So, this certainly is, one that I’m always wanting to talk about. So, I live in a suburban community that was at the time when my kids were in school and we moved here, you know, frankly, more affluent than we could afford you know, but that was where we needed to be to get the services that our kids needed. And I think that a lot of people are forced to make that choice.

[00:06:34] Charlie: But this particular piece was about the quality of. teachers in the environment of having a shortage of special education teachers. And I’ll tell you a short answer to that from my experience is that they ranged from horrendous to miraculous. My daughter in particular, had a terrible special education teacher who was really the biggest reason why we were forced to get an out-of-district placement and all the hiring of advocates and lawyers and people you can’t afford that goes along with that.

Alisha: Yes.

[00:07:04] Charlie:  On the other hand, several years later, after years of, of making more a lot of progress, my daughter decided very courageously, I thought that she was going to come back and do her senior year at the local high school. Now, it’s hard enough to do your senior year, come back for one year when you’ve been away for years. And I think the fact that it was in the middle of the pandemic makes it even, you know, made it even harder. But I have to say, in this case that ended up being very successful, largely because of the work by a team of special educators that really were nothing short of miraculous. You know, they were incredible, I’m happy to say, and thankful. So, when it comes to this issue of how do we solve this shortage, one of the answers here is, you know, what things that would be probably common sense in the non-education world.

[00:08:00] Charlie: Look, teaching is hard. I spent three and a half years as an adjunct doing it at the college level. And to do it well it became like an almost like a second full-time job. And I suspect very strongly that being a K-12 teacher is probably harder than that. But the fact is that I think being a special education teacher is harder still. I mean, that is a job, I know what it’s been like to parent two special needs students. I mean, for people who can do that all day, that is an unbelievably difficult job. And, you add up the fact of how difficult it is with the fact that there’s a shortage, I think that, in the real world, one of the answers would be we need to pay these people more. Yes, you know, but that sort of differential pay, it remains a very high wall to scale. And I just hope that, maybe one of these days we’ll be able to chip away at in a way that will help our students, in this case, some of our neediest students. So, that’s my thought on that issue.

[00:09:00] Alisha: I agree, I couldn’t agree with you more. I, too, in our family, we have two who have IEPs and, you know, all of the kids are different. All of them need things different, need different resources, and to your point, when I was superintendent, one of my long-term goals — which I never got a chance to get around to — was to train all of the teachers, including the general ed teachers, in special education, because the expertise that special education teachers have is phenomenal. And frankly, all of the students could benefit from it. But to your point, the work that specifically special education teachers do, I think is God’s work.

Charlie:  Oh my God, yes.

Alisha: You know, given the range of challenges that students present within classrooms. So, this shortage issue is a very serious one, as you pointed out, and it’s across the country. And it’s something that we have to address. Number one, because the students need it. And number two, you want to make sure that you’re in compliance with federal law.

[00:09:53] Charlie: Yes. Right.

[00:09:53] Alisha: Exactly. Thank you for that story, Charlie. I think we’re going to have a great show today. And so coming up, we are going to have Dr. Matthias Von Davier, who is the director of the TIMSS international test. We’ll be right back.

[00:10:31] Alisha: Dr. Matthias Von Davier is the J. Donald Monin S.J. professor in education at the Lynch School of Education and Human Development at Boston College and also serves as executive director of the TIMSS and PIRLS International Study Center. Prior to joining the faculty at BC, he held the distinguished research scientist position at the National Board of Medical Examiners in Philadelphia, Pennsylvania. He was a senior research director in the Research and Development Division at Educational Testing Service, also known as ETS, and codirector of the Center for Global Assessment at ETS, leading psychometric research and operational analysis of international large-scale assessments conducted by the center. He earned his PhD in psychology from the University of Kiel, Germany, specializing in psychometrics.

[00:11:19] Alisha: Welcome to the show, Dr. Von Davier. It’s wonderful to have you. I’m going to jump right in and ask the first question. So as your bio notes, you’re a German-educated psychometrician and researcher. So, can you talk with our listeners about your background, informative educational experiences, Tim’s and pearls and teaching education policy at Boston College?

[00:11:44] Dr. Von Davier: Sure. That you very much for the invitation and for your question. My background at least explains my accent, of course. Also, why I will be a little bit careful to talk a lot about countries I didn’t really grow up in. So, I teach at Boston College, mainly in psychometrics-related issues. So, we do a lot around methodology and quantitative methods, and I will really focus also on these types of issues and my responses naturally, even though educational policy, of course, gets lot of information from TIMSS and PIRLS, but I will talk about this in the context of our work in collecting and analyzing the data for TIMSS PIRLS in the international setting.

[00:12:29] Dr. Von Davier: But more directly to your question, so, my education includes college-level mathematics, computer sciences, and psychology. I have my PhD — was in quantitative methods and psychology. And I started to work at the Institute for Science Education while TIMSS 1995 was going on. I didn’t know about it, or not much about it back then.

[00:12:53] Dr. Von Davier: And I did my PhD there. Since 1997, I’ve been mainly in the U.S. I’ve been working at the Educational Testing Service in Princeton, New Jersey, pretty much since ‘97 and 2000, continuously, first as a research scientist and then ending up as a senior research director at ETS in the Center for Global Assessment, in the Center for Statistical Theory and Practice, and a few other places.

[00:13:22] Dr. Von Davier: And I’ve been working on NAEP, on PIAAC, which is the adult assessment by OECD, later on PISA. And I’ve been working pretty much all the time on TIMSS and PIRLS as the consultant to Boston College. So, I have a long-standing relationship with Boston College advising them on all kinds of statistical and psychometric analyses that they had to do.

[00:13:46] Dr. Von Davier: Then three years ago, I came to Boston College to lead the TIMSS and PIRLS International Study Center. And right now we’re working on PIRLS 26, TIMSS 23, and very soon, hopefully very quickly, also on TIMSS 27 because the cycle never ends. So, these are all cyclical assessments — and some other related projects. So, we have a really great staff there, the outstanding professionals, and we’re also lucky enough to get a lot of graduate students and PhD students who work with us in the center.

[00:14:18] Alisha: Very fascinating. Thank you for that. And so, psychometrics and standardized test development are often hotly contested topics in K-12 education. Hopefully you would agree with that. And so, could you explain what psychometrics is and how testing companies employ a wide variety of experts in this field to produce standardized tests that are both statistically valid and a fair way to measure student learning?

[00:14:43] Dr. Von Davier: Yes, certainly happy to answer this. I think it’s hotly contested. I fully agree with that, but also often not well understood, I guess. Psychometrics is the study of how we can quantify individual differences in a way that is fair, that treats everybody the same, regardless of background, reliable, but it gives you nearly the same or almost the same results if you would redo the test. Valid so that it actually measures what it’s supposed to measure and not just some randomly related other properties. In some ways psychometrics has similarities — and I say that also in my lectures — to sports ranking systems, whether you are ranking tennis players, chess masters, baseball teams, other types of players, even in massive online computer games, you find very similar methods applied. And if you look at that, if you look at tennis rankings, chess rankings, other types of sports, competitive sports, there is much less of being hotly contested to be heard. And they apply almost the same methods.

[00:15:51] Dr. Von Davier: So, there’s a lot of mathematics involved. Scores are derived in scientifically rigorous and defensible ways. But of course, we have to make sure that we really measure what we want to measure. So, there’s mathematics alone or the statistics alone doesn’t help, doesn’t do the full 100 percent of the job, I should say. It can only go so far. So, we need content experts. We need experts in assessment, in sampling. Also, awareness of assessment in content areas, experts in how to assess content and context variables, etc. So, it goes way beyond just the administering a test. Also, I need to point out, TIMSS and PIRLS do not give scores to individual students.

[00:16:35] Dr. Von Davier: We describe countries or groups within countries, and that involves many more steps that I really can’t talk here in full detail, but I’m very happy to talk a lot about this, and I usually do. We also take care of very carefully to make sure that we cover the curricula that are being taught in the countries, and talk a lot with countries about what is being taught and how it can be reflected in our assessments.

[00:17:03] Alisha: A lot goes into this, and it’s important for us to know that as we think about the reliability of these assessments. So, thank you for that. So, since 1957, when the Soviet Union successfully launched Sputnik 1, there’s often been a strong relationship between education policy, STEM, and military or space applications. Can you talk about how education policy has gradually transitioned its focus to economic competition, as well as democratizing equality of educational opportunity?

[00:17:34] Dr. Von Davier: Very interesting question. So — and I was reminded that my alma mater, the University of Kiel in Germany — they actually started the Institute for Science Education exactly because of Sputnik, just shortly after this historic event in 1957. I think the institute was started in 1958, actually. I think there’s, of course, a push to look at education also in terms of economic outcomes. However, TIMSS and PIRLS keep the educational focus and keep the focus on what is being taught in schools. What are countries agreeing on in terms of content?

[00:18:12] Dr. Von Davier: So, we really look at the curricula. I mentioned that in the answer to my last question. We do this work for not an economic organization like some other assessments. We do this work for the International Association for the Evaluation of Educational Achievement, the IEEA, and they have a clear educational focus.

[00:18:32] Dr. Von Davier: Thinking about how to improve education worldwide. This organization goes back to 1958, has legal status since 1967, and has made numerous contributions and grew the field of international large-scale assessment before actually these more economic focused assessments. GEMS 95, but really goes back to earlier mathematics and science studies, so the first and second mathematics and science studies.

[00:19:01] Dr. Von Davier: For example, I, and I will close with this one. We do not only test students in terms of how well they do in math and science or reading, but we also gather and collect a lot of context variables systematically. Our last report that you will find online at the TIMSS and PIRLS website, for example, focused on student wellbeing based on the PIRLS 2021 data. PIRLS is the reading assessment that essentially is the other big assessment in our center.

[00:19:31] Alisha: Thank you. Going back to A Nation at Risk in 1983 in the U.S. and internationally, there’s been growing knowledge and data about the relationship between K-12 educational attainment, And the global competitiveness among nations, could you talk about the global education data landscape and how international testing like TIMSS have greatly expanded our understanding of education performance across the world?

[00:19:58] Dr. Von Davier: Yeah, sure. So, if you allow me, I would stay with the international level because many of these things really also apply to not only the U.S. but also other nations. So, we really want to make sure that we do not prescribe any particular policy. We would like to provide information to policymakers, but don’t really want to get into the way of what they then do based on their further analysis.

[00:20:25] Dr. Von Davier: So, we provide one perspective on the role of education or educational outcomes. Policymakers take a lot more information into account and then come up with policies. Of course, I want to admit — and it’s a very important finding that we find over and over — that economic strengths and educational outcomes, achievement measures, TIMSS and PIRLS and other assessments, they do correlate at the country level. So, you will see economically stronger countries will tend to be higher scoring than economically more challenged countries or developing countries or countries that are at the threshold, low- and middle-income countries. The same is true, not only for the economy, but also for the transparency index that talks about transparent government is, how transparent the society is, many problems there are. So even there, it’s a very high relationship, more transparent society, according to these kinds of criteria are, the higher tend to be the average achievement in that country. However, you can also argue this is like the famous chicken and egg problem. We need well-educated children in the next generation to value what can be done in a country to improve society, to improve economic outcomes, to improve equity and fairness, to do this. At the same time, we want well-educated children, we need resources. So that also is easier than for countries that actually can provide more resources to education. So, it’s not an easy world to live in, but we really have to try to understand these complex relationships.

[00:22:06] Charlie: All right, Professor Von Davier, this is Charlie Chieppo here. Thank you so much for joining us. So, the COVID-19 pandemic has heightened concerns about the state of education in the United States and internationally. Based on the data, can you talk a little bit about educational performance among high-performing countries, low-performing countries, and how the U.S. compares on some global measures?

[00:22:30] Dr. Von Davier: Yeah, of course, this was a very frightening and tough time for everybody, and, we know and we can’t ignore the effects the pandemic had on pretty much everybody in the world. And also on education systems. In the short term, we saw a lot of negative effects and we also were afraid that this big experiment online education wouldn’t work so well. And we see fallout from that, of course. So, children couldn’t go to school. Parents had to make space and time for having their students, their children, sitting as students at home, all of a sudden.

[00:23:05] Dr. Von Davier: International assessments felt these effects this way. Either we had to postpone, or we had some countries who had to postpone other assessments, fully postponed often by a full year, and we, in particular, had to essentially extend the testing period and that also had some adverse effects of course. So, for that reason it’s also not so easy to talk about the fallout and how to measure this. There’s another reason and I just want to make this as a very simple kind of comparison, we don’t really have a before and after test. We didn’t know COVID was coming, so we couldn’t really test students before and after COVID.

[00:23:45] Dr. Von Davier: And also, we didn’t have any countries that were unaffected. There were maybe some were less affected, but everybody was affected to some extent. And there were some, of course, yeah, ideas about how long it would take. I think we are still learning here. There was a relatively recent article, on December 13, I think, in The New York Times, that pointed out some quite surprising, more recent results, but doesn’t fully fit to this narrative of sustained, long-term learning loss. Of some things I think we are on a rebound. Some people think that at least, and I think we need to make sure we carefully monitor, and we have to make sure that we have to take education seriously. But in the end, I’m an optimist. I think COVID was incredibly tough, and we lost too many lives, we lost too many opportunities to interact and to learn.

[00:24:38] Dr. Von Davier: I think really that education is resilient and so in many countries I really have high regard for all the teachers and all the school administrators who did an incredible job trying to do the best they could under the circumstances.

[00:24:53] Charlie: It’s funny, I listened to your response to the question, and ne of the things that strikes me is that we really, this continues to be a moving target, you know, as we continue to get more data on the impact and how long lasting it is and all those kinds of things, we keep having to kind of adjust on the fly based on what we’re learning. So, you grew up abroad and through your professional work, travels, research you’ve had the opportunity to see how other countries particularly some higher performing countries in East Asia, Russia, Germany, prepare their students to succeed in STEM. Could you discuss what TIMSS data from other countries tells us about teaching math and science and what perhaps American K-12 education policymakers might want to learn from or emulate?

[00:25:40] Dr. Von Davier: Yeah, that’s a very tricky question even though it sounds that this is exactly what these types of assessments are built for. And it’s tricky for a variety of reasons. So, we look at between country comparisons. How well do they do? You have, of course, the list of countries, the average achievement and everything. But we also look at systems-level information that goes beyond the pure just how well are they doing on average.

[00:26:10] Dr. Von Davier: We have measures of how wide the range is of achievement within the country. So, the variance, the variability of those scores, we have a lot about context data and we also have publications that talk about system-level information. So, we, as researchers, are very careful when it comes to trying to emulate other systems. There has been a lot of, I would say, TIMMS and maybe PISA tourism in the early cycles of those assessments. So, there were a few countries that were high on the list. And then policymakers from all kinds of other countries went there for professional meetings and information sessions. And I think everybody learned who was there that they cannot simply transplant those findings, because they have to go deeper, they have to learn a lot more about what is going on in the system? What does the population look like, for example, in terms of disadvantaged groups.

[00:27:13] Charlie: As you describe this, I’m thinking about the sort of frenzy. there was for a time about everybody going to Finland, finding out that, well, there’s a whole lot there that makes it completely different, you know, and as you say, not necessarily something you can just copy.

[00:27:29] Dr. Von Davier:  Exactly. And think just because of that, I think it’s really a smart move, of course, to get informed. One of the publications that we have is the so called TIMSS Encyclopedia, where we have country-level systems information, where countries can look at what other countries are doing. There’s of course a lot more information out there, but this is our little contribution to that. Then the other point is we really need to look deeper. It’s not only the average achievement and then looking at what they do in education. We also need to look at how the country is set up. So, what kind of training do teachers go through? What kind of supports do parents get? What happens in education policy there, at a really deep level, how different are school systems, are they driven centrally? There are so many different school systems out there, and to simply transplant something, say, a centralized school system, cannot be transplanted to a system with federal states. So, it’s a very tricky question, but a very interesting one, and I can just answer it. Yeah, so just to look into all the different publications where you find information about the different countries.

[00:28:37] Charlie: Right. All right. Well, finally international members like TIMSS highlight the relationship between education, skills and innovation. Can you talk about the wider learning loss, educational impact and financial implications that the COVID pandemic has had on global K-12 education and on competition between nations?

Dr. Von Davier:Yeah, I would like to say a lot about this, but I would also like to go back to a previous point. So, there is certainly learning loss, but we also don’t know the extent of the learning loss, just because we don’t have this before and directly before and after measures. We see on average that many, many countries in the last two bigger assessments showed a decline somewhat, but we also saw that some countries maybe saw less decline compared to other countries. The Secretary of Education actually pointed that out on December 5, ‘23 right now. He said, here’s the bottom line: At an extremely tough time in education, the United States moved up in the world rankings in reading, math and science, all three categories. These are measures. While, unfortunately, many other countries saw declines. So, everybody went down, but the U.S. less than other countries and actually in two domains, almost not at all. So, in reading and science, there was very little change. So, I would say some countries might be more resilient. Some countries might have been better prepared. We don’t know exactly what went on. This is why within-country studies are so important. We, for example, in our last assessment, we had questions about COVID, of course, because we collected during and after COVID, but the responses were so diverse that we are going to publish the responses just as they are. We are not trying to somehow group them or make statements about, okay, this is a good approach, this is not so much a good approach because there are so many different ways. There are so many different school systems. So, I don’t want you to feel discouraged by this, but it’s something you really need within-country studies. It’s very hard to generalize because every country had a different approach to how they handled COVID. Whether it was hybrid, that schools were open for half-sized classrooms and some students stayed home, others were in class. Some other countries had full closings by region, some others I think, you know, we should, we should be to stay in class. So, you really described it so wisely as a moving target. We still learn, but we can’t just say, okay, this can be transplanted. This is what we learned about COVID. This worked in that country, so for the next pandemic we know what to do. Unfortunately, it’s not that easy.

[00:31:32] Charlie: Yeah. You know, it’s funny as you mentioned that I’m thinking what comes to mind is two neighboring countries, Norway and Sweden, for example, who approached the pandemic so differently in terms of the policy responses to it. So that is certainly a very good point. This was great, Professor Von Davier. I really appreciate your being here. I suspect there are some of us here who might want to get you back to have you talk to us about sports rankings, but that’s a different question for a different day. Thank you so much for joining us we really appreciate it.

[00:32:09] Alisha: Thank you, Great conversation today.

[00:32:10] Dr. Von Davier: Thank you for the invitation. It was really nice. Thanks.

[00:32:33] Charlie: Well, that was a great interview,. really enjoyed that. And next I am going to read the Tweet of the Week, which is from the U.S. Holocaust Museum. In honor of International Holocaust Remembrance Day, Holocaust survivor and museum volunteer Ruth Cohen reflects on her family and urges us to learn from history and take action against antisemitism today.

[00:32:55] Charlie: So, you can go to the U.S. Holocaust Museum’s Twitter feed on January 26 and see that and with that, I will say once again, Alisha, it is just an absolute pleasure to get to do The Learning Curve with you. So, thank you very much. I hope we can do it again soon.

[00:33:12] Alisha: Absolutely. Looking forward to it. Looking forward to next week’s episode. We will have Professor Robert Norell. He is a chair of excellence in American history at the University of Tennessee and the author of Up from History: The Life of Booker T. Washington. See you next time.