In this episode, we talk to Kudo Maggi, the founder and president of Mainstream Research, about polling in the upcoming election in Saskatchewan. We discuss polling accuracy, sample sizes, and how polling is done in Canada.
00:00:00.000Hey everyone, Wyatt Claypool here. Ever since we had the British Columbia provincial election and the New Brunswick provincial election go by, I think there's been a lot more discussion around polling, polling accuracy, and how polling even works, especially in Canada where the field is far smaller than what is down in the United States.
00:00:20.840And I think the best person to bring on today to talk about this issue is Kudo Maggi, who is the founder and president of Mainstream Research, who I think has done a fantastic job polling in these last two provincial elections and looks like might get it right again in Saskatchewan.
00:00:38.440But I do tons of videos on polling, Kudo. I talk about it all the time, and I still get people skeptical about polling. Like, is it rigged? Are people just putting out numbers? Do they even know who they're contacting?
00:00:51.000And I think you would agree with me. There's polling firms that you maybe wouldn't look at too much, but compared to the U.S., I find Canadian firms polling in Canada tend to get it right, and especially you guys, who in these past two provincial elections got it really on side.
00:01:08.440Yeah, that's right. I think, look, both the Americans also have an association that does have very strict guidelines. It's called the APOR, the American Association of Public Opinion Research.
00:01:22.220Here in Canada, we have the Canadian Research Insights Council, of which we're a member.
00:01:26.020And so typically, the member firms that are part of Crick have to sign on to a certain standards of polling accuracy, following specific best practices, and then sort of also goes along with sort of communication practices in terms of saying,
00:01:47.260never say anything that your poll doesn't actually say or that it goes too far for what your poll says.
00:01:56.200Some people say, oh, this is a guaranteed majority or whatever the case may be.
00:02:01.840As you probably know, in the B.C. election, we published 14 riding polls.
00:02:05.520I always caveat those a lot and say, look, small sample size, greater chances of error.
00:02:11.420However, I think of the 14, 12 ended up pretty close.
00:02:18.780It's the nature of riding polls, smaller frame.
00:02:21.380So, you know, we do our best to sort of warn people that, you know, it's a snapshot and it can be, it can have sources of error.
00:02:31.440And especially in a province like British Columbia, where the leaders of the parties are different, the parties are different, there's tons of independents running all over the place.
00:02:40.920Right now, and it might change with recounts and the mail-in ballots and provisional ballots that are coming in that still need to be counted.
00:02:47.160You guys currently are literally right on the numbers.
00:02:50.36046 MDP, 45 Conservative, and two Greens.
00:02:53.600And if it goes a little bit off from that, that's not an actual error.
00:02:58.460I have people, when I've been talking about, because I'm more of a Conservative person, obviously, saying I think there's a 75% shot that Conservatives are going to win.
00:03:06.440I'll still have people come back later and say, hey, you said that they were going to win.
00:03:11.360There's always, you have to show up and vote, by the way, people.
00:03:15.040And the only big miss that there was in British Columbia, and it's not on anyone for missing it, was the Conservative appeal inside of Surrey.
00:03:24.100But that's not something you can poll, because I've talked about and showed some of the crosstabs to show what I'm talking about when I say that it's impossible to really poll Surrey, and it's just basically a giant question mark.
00:03:35.180You guys get different rates of people answering your polls, where basically polling is very white.
00:03:41.800The people who answer polls are pretty much across the board very white.
00:03:45.160And you guys, in most of your polls in the South Asian community, sometimes could only get 10 people to answer the poll, maybe 35, I think, on the bigger days.
00:03:53.500And, you know, that's why, if you followed along, Wyatt, that you probably know this, that towards the end, we, and this isn't the first time we've done this in a daily tracker, is we'll publish what we call a roll-up.
00:04:08.200It's a roll-up of all the samples from the beginning.
00:04:10.780So, towards the end, on that last day, we were reporting 11,000 and change sample sizes to be able to look.
00:04:18.560So, yes, it's for the general public information as well, for the modelers specifically, and for ourselves internally to look at and say, okay, here's the vote among the South Asian community.
00:04:32.260On any given night, you're correct, the sample is way too small.
00:04:36.380But when we look at that roll-up, we could tell that the South Asian community, you know, started behind and took a lead.
00:04:43.800When we look at the East Asian, the Chinese communities in B.C., we knew that it was an overwhelming support for the conservatives.
00:04:52.640And so that's why we could say comfortably that those riding polls that we did in Richmond, Richmond Center, Richmond Stetson, were more likely to be correct because of the population of Chinese in Richmond.
00:05:06.380Even at Richmond-Stevenson, the only reason the NDP effectively won that riding was because Jackie Lee was running as an independent on the basis that I am the more conservative option than the current conservative candidate.
00:05:19.580And that siphoned away enough of the votes.
00:05:21.400But you can't actually poll independents often.
00:05:23.900It's like back in Ontario in 2022, nobody was going to be able to poll that, what's her name, one Halderman, Norfolk, the provincial version.
00:05:45.820And I think that's why, you know, the models are going to be better in, the models were better in New Brunswick.
00:05:53.800The models were better in, will be better on Monday night in Saskatchewan because it's a very established, you know, established parties, established guidelines.
00:06:04.620You had the surge of the conservative party that we could, it's very hard to model that.
00:06:13.640And you had a riding redistribution all at the same time.
00:06:18.240Those both prevent for, you know, I'm not the modeler, Robert from my team is, but I knew that he was having those types of challenges and so did other modelers.
00:06:28.980But I still think, I'm going to wait until Monday night to see what exactly happens with those last 65,000 ballots.
00:06:40.080A lot of people are sort of analyzing and it's like, no, let's wait till Friday, which it is today.
00:06:48.700By the end of the day, according to Elections Canada, they've told me, they've written me specifically saying they're going to have the riding by riding number of ballots.
00:06:55.820And then we'll kind of be able to know, can some of these seats flip?
00:07:19.880It's just slightly on the side of the NDP.
00:07:22.700It's like by 20 votes, it's effectively a tie.
00:07:24.720So, yeah, it's very realistic for the ridings to flip at the moment.
00:07:30.080And two, the thing is, even if a bunch of these flip conservative and you guys had called the race 46 NDP 45 conservative, that's a margin of error.
00:07:39.400And that could be tons of explanations for why a poll is never going to get it precisely right.
00:07:46.540And you guys have to correct a lot for response bias.
00:07:48.960I was really happy to see this because other polls sometimes they just put out the numbers and it's just whatever the numbers were.
00:07:56.000And you guys will take especially university educated voters and usually weight them down because, you know, that somebody who has a degree in sociology, who's an engineer, somebody who went to took a master's degree, they're far more likely to pick up the phone and say, oh, they want my opinion.
00:08:11.600I'm going to give them my opinion rather than some guy who just got home as an electrician.
00:08:16.280He doesn't want to take a poll, even though IVR polls are actually pretty fast.
00:08:19.860Yeah, and we've been polling for education in most provincial and national polls now for the last, I'd say, since about 2018, just because, again, you know, the response rates are down, but also the type of people who answer are down.
00:08:36.500So you're having to make these adjustments, no different in the United States as well, but we've been doing polling there for the last year and a half.
00:08:44.120It's the same kind of adjustments that we have to make.
00:08:47.240Yeah, and that's the thing that you see a lot in the U.S. in the last few elections where oftentimes pollsters are completely off the results.
00:08:55.840And what I tell people is very often nobody's trying to confuse anyone or trick anybody by saying this guy's up 10 points and then he ends up losing.
00:09:04.100Oftentimes it's because the people taking the polls don't want to take the polls or they don't take them in the same rate.
00:09:09.360So what you hear a lot of, and you guys have been doing some U.S. polling, is that in the past two election cycles for president, they say that there's always a hidden Trump voter.
00:09:18.980Somebody who is definitely showing up for Donald Trump, but a pollster has a very difficult time being able to find that person when they pick up the phone.
00:09:27.020So you'll even have, and I follow the show of, I'm not sure if you know Richard Barris, the people's pundit, he's kind of like Janet Brown for the U.S.
00:09:36.020where he tries to balance education, work type and whatnot, so that he doesn't end up having crazy poll results in Georgia, in a county like Coffey County, where it looks like it's going Democrat, but it's because you accidentally pulled a bunch of school teachers.
00:09:50.500And obviously they're more likely to vote for the Democratic candidates.
00:09:55.460Yeah, and there's lots of examples of that.
00:09:57.220We learned a lot in this B.C. election about balancing between true rural and sort of semi-urban rural within interior B.C., and we're applying that to the Saskatchewan polls that we know.
00:10:15.600So balancing between cities, towns, and truly rural populations within these rural areas.
00:10:21.640And so most of our error in B.C. came all from the interior.
00:10:29.360We overestimated the B.C. conservatives in the interior.
00:10:35.880And I'm actually expecting to see tonight that a lot of these ballots that are out there, you know, the late ballots that came in, the ones that were already counted, I don't know where they're from.
00:10:48.360But the ones that arrived late, to me, it makes logical sense that they came from places that were further away.
00:10:55.200And that's where Kelowna, Interior, North Island, etc.
00:11:01.240That's why they arrived at the elections B.C. late, is because they were further away.
00:11:06.360And so I expect some of that is going to balance out in the poll, some of that error outside of lower mainland and islands.
00:11:16.700Because not a lot of people are going to be sending in a mail-in vote in Vancouver, Quilchina, or Vancouver, Yaletown.
00:11:24.080There's no point in really mailing in a vote in Yaletown.
00:11:26.540You can take the elevator down to the first floor of your building.
00:11:29.560And that's probably where the polling place is, because there's not many places to really set one up in an area where it's only like five square blocks or whatever in Yaletown there.
00:11:38.540So that's where the conservatives can feel decently, like, you know, optimistic in some of these ridings.
00:11:44.420Because in Burke Mountain, Coquitlam, you're probably going to have not only disproportionately a lot of mail-in ballots for that riding, but a lot of them coming from the Burke Mountain area, where, yes, they were postmarked on the 19th.
00:11:55.780But they got to a Canada Post distribution center, and then it took a few more days for them to actually get to B.C. elections.
00:12:02.320So then they could send them to the right area to be counted.
00:12:07.440And I think it's a good assumption, because you do see people where they see 49,000 ballots left to count.
00:12:12.740And then a few days later, it's 65,000.
00:12:15.020It's like, it's just things don't work as smoothly as people think, especially if people have sent in some mail-in ballots from Haida Gwaii.
00:12:22.240Those are not going to be getting to B.C. elections so that they know they have them on hand.
00:12:26.500And it's not like Canada Post is going to get on the phone and say, just letting you know, you got another 300 mail-in ballots from Haida Gwaii.
00:12:33.100That's not how Canada Post is set up to give B.C. elections their data.
00:12:38.780And this is another thing, actually, because I want to talk about Alberta a little bit and also Saskatchewan, because they're two very similar provinces.
00:12:46.120But Saskatchewan, in a certain sense, is even more obviously has a bigger urban-rural divide, is that people will look at a poll, especially out of Saskatchewan, and they'll see, like in your latest polls, oh, Saskatchewan MVP is up on the Saskatchewan party.
00:13:01.640And it's like, okay, but that doesn't actually mean that they're winning.
00:13:05.980That means that they could be doing really well in Regina and Saskatoon.
00:13:10.460But in the rural areas, that's where it's actually going to be very difficult to flip voters, because it's not like people are uniformly going to be going towards that party.
00:13:19.720Especially a lot of parties end up overplaying their hand to a specific community who's doing really well and then hurting themselves elsewhere.
00:13:27.0802023 in Alberta, I kept telling people, and I said this actually during the 22 leadership race for Danielle Smith, you have to understand the culture of Calgary, and it's not rural Alberta.
00:13:37.400And so you actually, and this is where I see now, people will see a poll come up for Alberta and say, Alberta UCP is up 5% on the NDP.
00:13:45.240And I'm like, that's bad, because Alberta and UCP barely won on seats with an 8.6% margin of victory on the popular vote.
00:13:54.240And that's the same thing with the NDP, but in reverse, where they're like, we're up by 5 points in the poll.
00:13:58.600But do you guys have the ability to dig into these rural ridings?
00:14:05.160Yeah, and I guess, you know, that was the challenge in Alberta.
00:14:09.360And, you know, if you look at our, say, last 10 provincial polls, going back to, we usually get the urban centres correct, and then where we miss is outside of those.
00:14:22.680And that can make a significant impact on the vote intents.
00:14:27.760What it doesn't make an impact on is seat counts.
00:14:29.700That's why I said to everybody, I don't even, the different polls that were out in BC that had these crazy different, some people had the NDP up in, outside of mainland and the island.
00:14:41.620And I said, it doesn't really matter if you model that one versus our poll that had a 20-point lead, it flips two seats.
00:14:50.800Whereas a one-point flip in census metro Vancouver flips a dozen seats.
00:14:57.600So, it's really much more important about getting it right.
00:15:01.580And that's why, you know, I was proud of the number that we got in Calgary in the last Alberta election.
00:15:07.640I, you know, I was famously called out for saying, I thought the NDP was going to pull it out.
00:15:13.260In the end, it came within a thousand votes.
00:15:16.620And our vote shared, again, the only error we had was from outside Calgary and Edmonton.
00:15:23.780Yeah, and no one can ever be blamed for getting it wrong when it's literally 1,200 votes across multiple ridings where people are losing and winning by seven votes and 50 votes and whatnot.
00:15:34.460And we even have that right now in Surrey with people leading with 107 votes, 96 votes, or Hwanda Buka Malahat, 20 votes.
00:15:42.180That's not really a margin that you can really count on.
00:15:45.060And in BC, too, really, I think what it came down to as well is that the money difference does matter.
00:15:52.040Your get-out-the-vote campaign really does matter because I think that's the only reason that we see the BC NDP leading a little bit when it comes to the popular vote is I was involved in the BC Conservative campaign.
00:16:03.720And this isn't like a secret I'm revealing.
00:16:05.380The riding campaigns were done with five bucks and a ham sandwich in some cases, even if it was a winning riding, because you don't have the experienced people.
00:16:15.120You don't have people who can go around chasing ballots in terms of finding people who haven't voted, figuring out a way to get them to vote.
00:16:21.200And so it really became a big infrastructure race.
00:16:24.940And I think it will be way easier to model, even though you're pretty dead on for this one.
00:16:29.280Obviously, the ridings are not always going to be the same of the ones that you predict might go blue, they might go NDP, and the NDP ones might go conservative.
00:16:40.120But even with the equal money between the two parties next time, with an actual baseline to base the party's performance off of, it will make everything way better.
00:16:49.840Because the BC Liberals were obviously more of a urban riding or an urban party, and then they would drop some of the more blue-collar rural areas to the NDP.
00:17:01.320And it seems like it's kind of inverted, where now it's almost like Alberta, where the BC NDP are very much an urban party, that urban-suburban party, and then the Conservatives are the suburban-rural party.
00:17:14.320Yeah, we often talk about that organizational advantage and who's going to overperform the polls.
00:17:19.980You know, consistently, like in Ontario, the PC has overperformed the polls.
00:17:25.700In a handful of provinces, the NDP outperformed the polls.
00:17:33.000Naturally, the Liberal Party of Canada traditionally has overperformed the polls until this, you know, really until late 2023, early 2024, where they've had that operational advantage.
00:17:45.640I think that's why I'd expect to see the SAS party actually do better on Monday than what the polls are saying, because it's the advantage of incumbency.
00:18:00.140There's generally a general desire out there for change, and I think it's right across North America, right across the Western world.
00:18:09.240But to think that Scott Mullen and the SAS party could lose to me is, it's, somebody said to me, oh, wouldn't it be, I was like, yeah, no, I've seen this before.
00:18:20.800I've seen it be close before, and it's just not going to happen.
00:18:24.060It's just practically speaking, the NDP's pitch is programs.
00:18:29.480A lot of NDP pitches provincially is always, we are the big programs party.
00:18:33.620We're the ones who are going to have the bigger social safety net.
00:18:36.620We're going to expand the public service and whatnot.
00:18:39.320And the problem is that that has a specific appeal that can work out well in Regina and Saskatoon and some of the other smaller cities in Saskatchewan.
00:18:50.740But then when you go out and it's just pure farmers, it's like, I don't know who that's then appealing to.
00:18:56.300And this happens all the time where somebody is chasing a voting block that doesn't actually carry them over the finish line when it comes to the full election, that you actually eventually need to go and find a way of figuring out which riding actually has flippable voters that's on the other side, rather than just winning what you're already winning even harder.
00:19:16.460And that's actually what was my criticism of the UCP's campaign in 2023, is a lot of the pitches made populists like them more, but not what I try as the nervous middle class voter who doesn't want, who isn't like moderate, like they want super moderate, mild policies, but they want you to talk about like taxes.
00:19:36.940And they want you to talk about economic health, they don't care about like, for example, the Alberta Bill of Rights type stuff.
00:19:44.120Yeah, or what are they called, the streaks in the sky or, you know.
00:19:51.580Yeah, the chemtrails, it's like, it's not a vote getter.
00:19:55.740Here's actually a great example of the US.
00:19:58.760People don't realize how much on people watching politics online skews things badly on the ground.
00:20:04.920People don't actually know that the average voter doesn't care about online discourse.
00:20:11.200Tulsi Gabbard, for some reason, is now a big darling of like the right in the US, even though she's actually not that conservative on many, many issues.
00:20:20.180Like, but everyone acts like this is a big get.
00:20:22.180I'm like, she got destroyed in the Democratic 2020 primary, because nobody cares about foreign policy.
00:20:27.900And when you become the foreign policy candidate, you're never going to pull more than like one or two percent, because the average voter doesn't care about the things that people online care about.
00:20:36.960And that's the problem I see that certain conservatives or like the PPC or the Green Party tends to fall into.
00:20:43.300They think that the hot button online issues are the issues that are going to cause somebody who just drove their kids to soccer practice, grabbing like going against Starbucks and like shops at Safeway and just watches like entertainment Netflix shows.
00:20:58.440They don't care about your like really oddly specific foreign policy issues or populist type issues.
00:21:06.620And this goes for the left and the right, the NDP, the conservatives, you know, the liberals.
00:21:11.140They all have these terminally online issues that end up making them completely miss the average voter.
00:21:17.100And I think that that's why polling actually does matter.
00:21:19.760I also encourage people, I'll throw a link in the description of this video below for anyone who wants to sign up to get the detailed breakdowns of Main Street research polls, because I think it's really good.
00:21:28.580And it makes you better at analyzing politics to actually know what average people think of issues.
00:21:34.740You guys don't really do issue polling, but just knowing which demographics are...
00:21:38.760We do. I mean, we do a lot of early issue polling.
00:21:43.080You know, we don't do a lot of that publicly.
00:21:45.120We do a lot of it for, you know, our clients.
00:21:50.680We have a lot of issue clients that we ask questions for as part of our omnibus questions.
00:21:57.220And then, you know, yeah, some of that informs the polling.
00:22:01.960I mean, you know, this election, all these elections across Canada, the last three, at least potentially four, have all been about the same thing.
00:23:05.160And I actually think whoever sort of successfully sells himself as the incumbent, as the non-incumbent and the change maker, that that's who's going to win.
00:23:18.080Right now, I'd give the advantage to Trump.
00:23:21.900And so another thing, actually, I think you just touched on that people need to know just about the polling industry is that it's not like you are paying.
00:23:31.100People think like, oh, this poll has the liberals doing a little bit better.
00:23:43.720Most polling is done for free to prove that you're good at polling.
00:23:47.800The whole point is that it's a very easy thing that actually gets checked because there's an election that proves whether or not that you are good at it or not.
00:23:56.060So you can do market research later for other people and you may know that you're not good at it.
00:24:00.040And, you know, that's why we do it with, you know, we've done these daily trackers and subscriptions.
00:24:05.400And, you know, and the subscriptions keep us going.
00:24:11.820We've been doing it pretty successfully in the United States as well.
00:24:15.420A subscriber model, you know, that informs the public because otherwise you end up having two classes of people.
00:24:24.020You have the politically informed people who can afford real quality polls and then you have everybody else.
00:24:32.200And, you know, look, I get along okay with some of the aggregators, PJ Fournier and Grenier and stuff.
00:24:43.300But the prevalence of those folks in the media has actually caused less understanding of polling, not more, I believe, because...
00:24:54.200And some of the ways they report things, I'm not a big fan of the whole plus one, plus two.
00:25:00.360Here in Canada, especially, there was a lot of comments on social media saying, oh, you guys were the worst off.
00:25:06.000You had the conservatives leading plus three and the NDP won.
00:25:10.700Not understanding sort of like how math works.
00:25:13.780You know, I get annoyed at it because it's their fault.
00:25:20.340They didn't realize that when other firms said plus five NDP or plus four NDP, they were also predicting like A, C, White, or two because their regionals were off.
00:25:32.520And there is an idea that only one of those things that you kind of feel, too, that when you look at a poll, I can kind of feel that this stands to reason that this makes sense.
00:25:43.800That if I see an Alberta poll pop up, like happened a lot during the provincial election, and somehow the rural areas, Lethbridge, Redbeer areas are tight, they're not tight.
00:25:56.420If you go around talking to enough people, you can kind of get the mood and you can see which polls are working well and which ones are not.
00:26:08.680The problem is, is that you can't mix bad polls with good polls and pretend that you're now going to get a good result simply because we've added more polls together.
00:26:16.680Again, there were Alberta pollsters who missed the popular vote in that election by 14 and a half percent or 13 and a half percent.
00:26:24.260And it's like, that's, and usually it's a strategy firm that does that because strategy firms are in the business of working for a party usually and giving advice.
00:26:31.880The analogy that I always use, Wyatt, in that is that, you know, you can't take two darts, shoot it at a dartboard, miss three feet to the left on the wall with one dart and three feet to the right of the dartboard with the second dart and call it a bullseye.
00:27:11.140What do you think was the big difference or is the big difference in terms of the accuracy and quality between online polls and IVR?
00:27:18.980Because you guys are famously one of the few IVR pollsters and you use what you call smart IVR.
00:27:25.240But most people just use online panels these days.
00:27:28.340What you find is the reason that you don't use online and other people do.
00:27:31.960So, there's been really, really good online pollsters.
00:27:38.040Not all online polls are the same, I would say.
00:27:41.440I would ask, I would say, you know, I have a great deal of respect for Lege and a few others that I know do probability recruited panels.
00:27:51.940And I think that that's one of the things that people need to keep an eye on where it's like, how was this panel recruited if it's just because they have been, there's been a few studies about them being hacked by AI and bots and other things.
00:28:06.040So, I think probability recruiting for online panels.
00:28:10.180And we do use some of those in the U.S. in a blend with IVR.
00:28:16.940But traditionally, I like to stick to 100% probability sampling because, well, usually nothing's ever happened like this election in BC where we saw all the polls, the different types of polls kind of converge very late.
00:28:32.760Because usually it's as we get one or two weeks out from an election, there might be difference early on in the election or pre-election, but you usually see convergence.
00:28:43.500And it's more has to do with informed vote, vote that's paying attention and informed vote versus vote that isn't paying attention.
00:29:47.440If not, then I believe that this will probably be the end of the episode here.
00:29:52.020I caught him in the airport, so he was, like, going to Miami or something like that.
00:29:56.420And he was willing to take, like, an interview on his phone, so that was good to see.
00:30:01.820But anyways, maybe what I'll do is I'll pause it here, and if he doesn't come back, then I'll bring him back in just a quick second.
00:30:07.860Or I'll just come back on and say, hey, that's the end of the episode for realsies this time.
00:30:11.460Okay, it appears that that was the end of the interview.
00:30:15.280Thank you for watching this video, even though it was a little bit shaky at the end there.
00:30:20.380I hope you learned something, feel more informed about how polls work.
00:30:24.180And I'd probably love to have Kudo on in the future, especially in the middle of an election campaign, one that I'm not working on,
00:30:31.500because that would have looked really bad if Kudo came on to talk about polling and I'm just, like, trashing the NDP in my videos or whatever.
00:30:38.460So that's why I specifically wanted to do this after the election campaign was over, because I think that's a more fair way of doing stuff.
00:30:44.860But if you want to support the show, you guys can donate to the Give, Send, Go link in the description below or pinned at the top of the comments underneath.
00:30:53.000And also, if you want to sign up on my website, if you ever want recommendations for nomination race candidates in your riding, sign up at wyattclaypool.com.
00:31:04.920And what it allows me to do is if I know there's a really good nomination candidate for a conservative party in your area,
00:31:11.080I can basically email anyone in those postal codes and let you know that these people are actually solid or give you, like, a ranking of how I would vote in that race.
00:31:20.320Or maybe I can even make small videos for those races and then put them out privately to those who live in the areas.
00:31:27.580Anyways, that should be it for me today, guys.