In this episode, I sit down with journalist and author Renee Vellian to talk about her new book, "The Dark Side of the Internet: How Russia Uses Social Media to Manipulate Us, Control Us, and Control the World," and why she thinks we should all be worried about what s going on in the world of social media. She talks about how the internet is vulnerable to manipulation by anyone, and what we can do to stop it. We also talk about how she and her co-author, Alex Vatanka, got their start in the field, and why they think we should be worried. This is a really important episode for anyone who wants to know what's going on with social media and how it affects us, and how we need to be prepared to fight back. It's a must-listen episode, and you won't want to miss it! Thanks to Renee for coming on the show, and for being kind enough to share it with us! Thank you so much Renee, and thank you for being brave enough to take the time to share her story with us. I really appreciate it. And I hope you enjoy this episode. -Jon Soraya, and I know you'll agree that it's one of those rare moments where you'll be blown away by what's coming out of your mind! -Sam Harris and I can't wait to do it again! (and again, thank you, again, for listening to this episode of The Dark Side Of The Internet. . - Jon Soraya and I'll see you again next week for listening and sharing it with the rest of your support. --Jonah, Jonah, and Jonah Jonah's podcast, Caitlyn, and Sarah, and Johnathan, and all of us at the podcast, and we're looking forward to hearing back from you, and your support, and support you in the next episode! -- Thank you Jonah and I appreciate you, Sarah, again and all your support and support, all of your feedback, and so much more! <3 -Jonah and Sarah -- Jonah & Sam Harris, and much more -- thank you. , Jonah is Sarah, Sarah -- Thanks Jonah: -- Sam Harris: Thank you for listening? -- Sarah: Sam Harris's podcast:
00:00:23.000How did you start researching these online Russian trolls and bots and all this jazz?
00:00:30.000Yeah, so a couple years back, in around 2015, I had had my first baby in 2013, and I was getting them on these preschool lists.
00:00:38.000And what I decided to do was I started looking at anti-vaccine activity in California because I had a kid and I wanted to...
00:00:46.000You know, put them on preschool lists where I was going to fit with the parents, basically, as someone who vaccinates.
00:00:51.000And I started looking at the way that small groups were able to kind of disproportionately amplify messages on social channels.
00:01:00.000And some of this was through very legitimate activity, and then some of it was through really kind of coordinated, deliberate attempts to kind of game ways that algorithms were amplifying content, amplifying particular types of narratives.
00:01:13.000And I thought it was interesting, and I started writing about it.
00:01:16.000And I wound up writing about ways in which hashtag gaming, ways in which people were kind of using automation to just be in a hashtag all the time.
00:01:26.000So it was kind of a way to really gain control of shared voice and what that meant when very small groups of people could achieve this kind of phenomenal amplification and what the pros and cons of that were.
00:01:36.000And then this was 2015. So the way that this sort of...
00:01:59.000To really kind of own a narrative, really push this brand, this digital caliphate, to kind of build it on all social platforms almost simultaneously.
00:02:07.000And the ways in which information was hopping from one platform to another through kind of deliberate coordination and then also just ways in which information flows kind of contagion style.
00:02:18.000And I wound up working on thinking about how the government was going to respond to the challenge of terrorist organizations using American social platforms to spread propaganda.
00:02:30.000So what we came to realize was that there was just this information ecosystem and it had evolved in a certain way over a period of about eight years or so.
00:02:39.000And the kind of unintended consequences of that.
00:02:42.000And the way that Russia kind of came into the conversation was around October 2015, when we were thinking about what to do about ISIS, what to do about terrorism, and terrorist, you know, kind of proliferation on social platforms.
00:02:57.000This was right around when Adrian Chen had written the article, The Agency for the New York Times.
00:03:05.000The first time an American journalist had gone over there and actually met the trolls, been in St. Petersburg, and began to write about what was happening over there and the ways that they had pages that were targeting certain facets of American culture.
00:03:18.000So while we were in D.C. talking about what to do about terrorists using these platforms to spread propaganda...
00:03:25.000There were beginning to be rumblings that Russian intelligence and, you know, Russian entities were doing the same thing.
00:03:31.000And so the question became, can we think about ways in which the internet is vulnerable to this type of manipulation by anyone, and then come up with ways to stop it?
00:03:42.000So that was how the Russia investigation began, was actually around 2015, a handful of people started looking for evidence of Russian bots and trolls on social platforms.
00:03:53.0002015, if we think about social media and the birth of social media, essentially, it had only been alive for, I mean, was Twitter 2007, I believe?
00:04:06.000Like, eight years of social media, and then all of a sudden they figured out how to game this system, and then they figured out how to use this to make people argue against each other.
00:04:18.000Yeah, so I think, so there was this, if you go back to like, remember like GeoCities?
00:04:29.000So there have always been, you know, kind of, the thing that was great about the internet, like internet 1.0 we can call it, right, was this idea that everybody was given a platform and you could use your platform, you could put up your blog, you could say whatever you wanted.
00:04:42.000You didn't necessarily get attention, but you could say whatever you wanted.
00:04:45.000And so there was this kind of consolidation as social platforms kind of came into existence.
00:04:51.000Content creators were really excited about the fact that now they not only had this access to write their own stuff, but they also had access to this audience because as the network effects got more and more pronounced, more and more people came to be on social platforms.
00:05:06.000And it originally wasn't even Facebook.
00:05:07.000If you remember, it was like, you know, there's like Friendster and MySpace and social networks kind of evolved.
00:05:12.000When I was in college, Facebook was still limited to like, you know, a handful of like Ivy League schools.
00:05:18.000And as you watch this consolidation happen, you start to have this information ecosystem really dominated by a handful of companies that grow very large.
00:05:29.000Because they're providing a service that people really want.
00:05:32.000But there's a kind of mass consolidation of audiences onto this handful of platforms.
00:05:36.000So this becomes really interesting for regular people who just want to find their friends, reach people, spread their message, grow an audience.
00:05:45.000It also becomes really interesting for propagandists and trolls and, in this case, terrorist organizations and state intelligence services.
00:05:52.000Because instead of reaching the entire internet, they really just kind of have to concentrate their efforts on a handful of platforms.
00:05:58.000So that consolidation is one of the things that kind of kicks off one of the reasons that we have these problems today.
00:06:18.000Because you can see it's an actual person.
00:06:19.000If you're narrating something, you know, if you're in front of the camera and explaining things, people are going to know that you're an actual human being.
00:06:30.000Whereas there's so many of these accounts that I'll go to like I'll watch people get involved in these little online beefs with each other and then I'll go to some of these accounts like this doesn't seem like a real person and I'll go and it's like hashtag MAGA there's a American Eagle in front of a flag and then you you read their stuff like wow this is this is probably a Russian troll account and it's strange like you feel like you're not supposed to be seeing this like you've seen the wiring under the board or something and then you'll go through the timeline I think?
00:07:14.000Yeah, so in 2016, there was a lot of that during the presidential campaign, right?
00:07:20.000And there was so much that was written, you know, we can go back to the free speech thing we were kind of chatting about before.
00:07:26.000There was so much that was written about harassment and trolling and negativity and these kind of hordes of accounts that would brigade people and harass them.
00:07:35.000Of course, a lot of that is just real Americans, right?
00:07:37.000There are plenty of people who are just assholes on the internet.
00:08:20.000But sock puppet is the other way that we can refer to it, a person pretending to be somebody else.
00:08:26.000So you have these sock puppets, and they're out there, and they're tweeting in 2014 about the Russian annexation of Crimea, or about MH17, that plane that went down, which Russia, of course, had no idea what happened, and it wasn't their fault at all.
00:08:38.000And gradually, as they begin to experience what I imagine they thought of was success, that's when you see some of these accounts pivot to targeting Americans.
00:08:49.000And so in late 2014, early 2015, you start to see the...
00:08:55.000This strategy that for a long time had been very inwardly focused, making their own people think a certain way or feel a certain way or have a certain experience on the internet, it begins to spread out.
00:09:07.000And so you start to see these accounts communicating with Americans.
00:09:11.000And as we were going through the datasets, which the Twitter dataset is public, anyone can go and look at it at this point, you do see some of the accounts that are kind of, you know, that were somewhat notorious for being really virulent, nasty trolls, anti-Semitic trolls going after journalists,
00:09:32.000Being revealed as actually being Russian trolls.
00:09:36.000Now, it doesn't kind of exculpate the actual American trolls that were very much real and active and part of this and expressing their opinion.
00:09:44.000But you do see that they're mimicking this.
00:09:46.000They're using that same style of tactic, that harassment to get at real people.
00:09:51.000And if they do get banned, if their account gets banned, they just simply make another account.
00:09:56.000They use some sort of a, you know, what is it, a virtual server?
00:10:45.000Yeah, so what they're doing is they're operating in communities.
00:10:49.000So one of the really common criticisms of, you know, people who, a lot of people think that this didn't have a huge impact, didn't, you know, did it swing the election?
00:10:59.000But what it does do in the communities that it targets is it can change that tone.
00:11:04.000And that's where you see – I mean, I think everybody's probably had this experience.
00:11:10.000You're part of a group and then a new person gets added to the group and the dynamic changes.
00:11:13.000It's very much the same kind of thing, just that these are not real people who are joining the group.
00:11:18.000And so there's this opportunity to – Kind of expand the bounds of tolerance just that little bit more or try to normalize using particular ways of communicating that maybe a group wouldn't naturally gravitate to,
00:11:36.000So there are definitely ways in which any type of troll doing this, doesn't have to be a Russian troll, has this ability to kind of shift the language, shift the community, shift the culture just a little bit.
00:11:53.000Do we have someone who's ever left there or become a whistleblower who can give us some information about what the mandate was and how it was carried out?
00:12:03.000There have been a couple of whistleblowers and actually some investigative journalism in Russia that's covered this.
00:12:09.000They describe the employees of the Internet Research Agency.
00:12:13.000So it's a little bit like a social media marketing agency, plus tactics that we would not expect a social media marketing agency to use.
00:13:18.000And then the other thing that they do is they talk about in Mueller indictment, you see some really interesting descriptions of, like, the stand-ups that they have.
00:13:24.000Stand-up is a thing you do at a tech company where everybody kind of stands up and talk about your goals and responsibilities and blockers and things.
00:13:30.000And in these stand-ups, they would be sitting there saying things like, if you're targeting black LGBT people, make sure you don't use white people in your image, in your meme, because that's going to trigger them.
00:13:42.000So trying to get at the very niche rules for communicating authentically in an American community.
00:14:09.000The degree of granularity that they have to recognize that if you are running a black LGBT page and your meme is of white people, you're going to cause some tension and consternation.
00:14:20.000And assuming that that's not necessarily what you want to be doing, you should go find the meme of black LGBT people to put as your meme for the day.
00:14:31.000There's a lot of understanding of American culture.
00:14:34.000And then there's a lot of understanding of trolling culture, and so these things combine to be a rather effective, you know, very effective social media agency.
00:14:43.000And is there an overwhelming sort of narrative that they're trying to pursue, that they're trying to push?
00:14:49.000So what we saw, so I did some of the research for the Senate, and the Senate data came from the platforms.
00:14:57.000So what I had was the attribution was made by the platforms.
00:15:00.000It wasn't like Renee deciding this was IRA. It was the platforms giving it to our government.
00:15:09.000Information in there, what it showed was that across all platforms, across Twitter, across Facebook, Instagram, YouTube, they were building up tribes.
00:15:20.000So they were really working to create distinct communities of distinct types of Americans.
00:15:25.000And that would be, for example, there's an LGBT page that is very much about LGBT pride.
00:16:07.000It read like a young woman talking about crushes on actresses and things, actually.
00:16:14.000It was really, besides the sometimes wonky English, virtually indistinguishable from what you would read on any kind of young, millennial-focused...
00:16:25.000It wasn't, none of it was radical or divisive.
00:16:31.000It wasn't like, the way that they got the division across was they built these tribes where they're reinforcing in-group dynamics.
00:16:50.000So both old far-right, meaning people who are very concerned about what does the future of America look like, and then young far-right, which was much more angry, much more like trolling culture.
00:17:02.000So they recognize that there's a divide there, that the kinds of memes you're going to use to target Younger right-wing audiences are not the same kinds of memes you're going to use to target older right-wing audiences.
00:17:11.000So there's a tribe for older right-wing, younger right-wing.
00:17:15.000In the black community, there's a Baptist tribe.
00:18:25.000So like me and you, we're having a conversation.
00:18:27.000We're developing a relationship on this page over time.
00:18:30.000And then I say, like, as this kind of person, we don't believe this.
00:18:35.000So it's a way to subtly influence by appealing to an in-group dynamic or appealing to, like, as members of this tribe, as LGBT people, of course we hate Mike Pence.
00:18:45.000As black people, of course we're not going to vote because, you know, we hate Hillary Clinton because we hate her husband.
00:20:08.000I got this data set and I was going through these Instagram memes and 133,000 of them.
00:20:17.000And I was, there was a cluster of images of Kermit the Frog.
00:20:24.000I was like, what the hell is Kermit the Frog doing in here?
00:20:27.000And so I, so then I go, so this, the way the platforms provide the data is I got like a CSV of the posts and then I got a folder of the images.
00:20:35.000And so in order to like connect the dots, I had to have the image up on one screen and the, this thing, the CSV up on the other screen.
00:20:45.000And we turned it into a database that we could track things a little bit more easily across the platforms.
00:20:53.000So I have this cluster of Kermit the Frog memes and I go and I look and I realize that they're attributed to an account called Army of Jesus.
00:21:29.000And all of a sudden, the data set turns into Homer Simpson memes.
00:21:34.000So again, like this kind of raunchy Homer Simpson culture.
00:21:39.000And again, it's attributed to Army of Jesus.
00:21:41.000And then I go through all this and realize that they didn't get to actually making Army of Jesus a Jesus-focused page until like 900 posts in.
00:21:49.000So they just renamed the account at some point.
00:22:08.000And then all of a sudden, the likes and things start pouring in.
00:22:12.000So what they're doing is they're actually like either deliberately or they're just creating placeholders.
00:22:19.000It's kind of a red flag when a brand new account that was created yesterday suddenly starts talking about some highly politically divisive thing or whatever.
00:22:27.000But if you lay the groundwork and you do it over a period of two years, then somebody who goes and checks to see what the account was, where it came from, how old it is, is going to see something that was two years old.
00:22:38.000So it's an opportunity to create almost like sleeper accounts where you create them now and then you activate them, you politicize them, you actually put them to use a couple of years in the future.
00:22:49.000So we saw all kinds of, we saw this over and over again.
00:22:53.000There was a Black Guns Matter account that turned into an anonymous account at one point.
00:22:59.000They were pretending to be anonymous, you know, the hacktivist.
00:23:02.000So they repurposed this Black Guns Matter page, which just had, it was advocating that black people buy weapons and carry, and it's like a pro-Second Amendment page, but for the black community.
00:23:13.000And they took that page, when it wasn't getting, I guess, a ton of engagement, and it became, it was called, oh gosh, I don't remember the exact name of the anonymous page, and I don't want to say it was something that's legit, but they pivoted into an anonymous page.
00:23:29.000And when they do that, do they go back and repurpose the content of the earlier posts?
00:23:39.000We didn't get that information from the platforms.
00:23:41.000There was a lot of stuff that I would have loved to have more insight into.
00:23:46.000We could see, again, you know, you'd think if you started following an Army of Jesus page and you had all this raunchy Kermit shit from like a year ago, that would raise some flags.
00:23:53.000I would assume that they scrubbed it and restarted, but I don't know.
00:23:57.000Your podcast with Sam changed how I look at a lot of the pages that I actually follow, because I follow some pages that have classic cars or something like that, and then I'll see them, and most of it is just photographs of cars, like beautiful old cars,
00:24:13.000and they'll have a giant following, and then all of a sudden something will get political.
00:24:21.000Like, this is probably one of those weird accounts.
00:24:24.000Like, they're getting people to get engaged with it because it represents something that they're interested in, like classic muscle cars.
00:24:32.000And then they use it for activism and they use it to get this narrative across.
00:24:38.000I think, I mean, I've seen it happen with some of mine, too.
00:24:42.000I think one of the challenges is, like, You want people to be aware that this stuff exists, but you don't want them to be paranoid that it's everywhere.
00:25:10.000It's this balance between when you make people aware of it, and I think people deserve to be aware of it, they deserve to understand how this plays out.
00:25:18.000The flip side of that is you do wind up in these weird, you know, you see it happen on social media now or click into a Trump tweet and you'll see like, you're a Russian bot!
00:26:26.000Or you get caught and then there's a, you know, until there's some confidence in the ability of platforms to detect this stuff, there's real concern among everybody that you're encountering something fake.
00:26:41.000Now, the overwhelming narrative is that the Russians were very much invested in having Trump win.
00:26:51.000And if they were very much invested in having Trump win, was the reason why they focused so heavily on the African American community?
00:26:58.000Because the African American community traditionally seems to vote Democrat.
00:27:02.000So they were trying to do something to break that up or trying to do something to weaken the position of the incumbent or Hillary Clinton and maybe put some emphasis on Jill Stein or some alternative candidates.
00:27:17.000Yeah, so the way that the political campaign, the political aspect of it played out, so they established, they started building these relationships in 2015. And, you know, they're doing this tribal thing, we've got our in-group, we're part of this community.
00:27:31.000And then what you start to see them do is early, they're actually, there was a tiny, tiny cluster in the early primaries where they were supporting Rand Paul.
00:27:41.000And then they pivot to Trump pretty quickly.
00:27:44.000And probably Rand Paul just didn't poll well and they were like, there's no way to get any lift here.
00:27:49.000But maybe Trump was getting, you know, some actual lift in the media.
00:27:54.000And so you see them move into supporting Trump.
00:27:56.000And then for the remainder of the data set from 2015 through the end, which was mid-2017 or so is when this thing ends, It's adamantly pro-Trump on the right.
00:28:07.000And on the right, you see not only pro-Trump, but you see them really working to erode support for mainstream or traditional Republicans, traditional conservatives.
00:28:18.000You see a lot of the memes about like, are you with the cuck-servatives or the conservatives?
00:28:23.000And so the cuck-servatives, of course, they've got pictures of Lindsey Graham and John McCain.
00:28:35.000Well, I think that they, you know, one of the theories is, and I believe this is probably true, they really strongly disliked Hillary Clinton because there was concern that she would, you know, things that she was saying about increasing freedoms in Russia were very threatening.
00:28:50.000They thought the best bet to get sanctions removed was Trump.
00:28:53.000So they had specific outcomes that they were hoping for, and that was one of, you know, so there's always like a political motivation.
00:29:01.000So there is this narrative around they just want to kind of like screw with American society, create divisions, amplify divisions.
00:29:07.000When you look at the political content, the clear and sustained support for Trump, and even more than that, the clear disdain for Hillary Clinton, there is not, on Facebook and Instagram, there was not one single pro-Hillary post.
00:29:44.000Because then they're using Bernie Sanders as a way to say this was stolen from him by the evil Clintons or Jill Stein.
00:29:52.000You know, here's a true independent, real liberal.
00:29:55.000We should be voting for her if we want to support a woman.
00:29:57.000So there are these feminism pages really pushing this narrative of Jill Stein.
00:30:01.000So you have the left-leaning pages, totally anti-Clinton.
00:30:05.000And then you have the right-leaning pages, staunchly pro-Trump and also strongly anti-Cruz, anti-Rubio, anti-Lindsey Graham, basically anti-Trump.
00:30:14.000Every, now what's called, establishment Republican.
00:30:17.000And there's this kind of pushing of people to opposite ends of the political spectrum.
00:30:24.000So this is where you get at the conversation around facilitating polarization.
00:30:31.000It wasn't enough to just support Donald Trump.
00:30:34.000It was also necessary to strongly disparage the kind of traditional conservative moderate center right in the course of amplifying the Trump candidacy.
00:32:50.000It seemed different to me than this one.
00:32:52.000This one seemed like We had moved into another level of hostility that I'd never experienced before, and another level of division between the right and the left that I'd never experienced before.
00:33:04.000And a willingness to engage with really harsh...
00:33:12.000Nasty comments, and just to dive into it, you would see it all day.
00:33:17.000I mean, there were certain Twitter followers that I think are pretty much human beings, but I would follow them, and they would just be engaged with people all day long, just shitting on people and criticizing this and insulting that, and it seemed like It seemed dangerous.
00:33:35.000It seemed like things had moved into a much more aggressive, much more hostile and confrontational sort of chapter in American history.
00:33:45.000If this is all done at the same time that this is happening, how much of an influence do you think this IRA agency had on all this stuff?
00:33:58.000That's the question that we would all like the answer to, and I unfortunately can't give it.
00:34:08.000The thing that we don't have, that nobody who looks at this on the outside has, is we can't see what people said in response to this stuff.
00:34:17.000So I've looked at now almost 200,000 of these posts, is what I spent most of last year doing, was this research.
00:34:26.000And we can see that they have thousands of engagements, thousands of comments, thousands of shares.
00:34:33.000We have no idea what happened afterwards, and that's the problem.
00:34:37.000So once the stuff comes down, it's really hard to go back and piece it together.
00:34:42.000So I can see that there are some, per your point, the really, really just fucking horrible troll accounts that they ran.
00:34:51.000They didn't necessarily have a lot of followers, but you see them in there like adding people.
00:34:55.000So they're, you know, at and then the name of a reporter at the name of a prominent person.
00:35:00.000And so they're in there kind of like draft on the popularity of, you know, famous people basically.
00:35:05.000And they're just saying like horrible shit.
00:35:11.000And one thing that was interesting with a couple of them is if you go and you look at their profile information, which was also made public, they would have a Gab account in their profile.
00:35:22.000So it was a remarkable piece of the culture in which you see that they're actually sitting on Gab too.
00:35:32.000And so they can also go and they can draw on.
00:35:55.000When this happens on a Facebook page, and they're doing something like telling black people not to vote, as black people, we shouldn't vote.
00:36:07.000And that's the piece that we don't have.
00:36:09.000So when we talk about impact, a lot of the impact conversation is really focused on, did this swing the election?
00:36:16.000We don't have, nothing that I've seen has the answer to that question.
00:36:20.000The other thing is, but the second question, the thing, when I think about impact, I think you and I agree on this, it also matters how does this change how people relate to each other.
00:36:32.000And we have no real evidence of, no information on that either.
00:36:37.000This is the kind of thing that lives in some, you know, Facebook has it.
00:36:48.000Yeah, so there were, here are my little stats here because I don't want to give you the wrong data.
00:36:53.000There were 10.5 million tweets, of which about 6 million were original content created by about 3,800 accounts.
00:37:02.000There were about 133 Instagram accounts with about 116,000 posts and then 81 Facebook pages and 17 YouTube channels with about 1,100 videos.
00:37:18.000And so they got about 200 million engagements on Instagram and about another 75 million or so on Facebook.
00:37:26.000Engagements are like likes, shares, comments, reactions, you know.
00:37:32.000So, it's hard to contextualize what we think happened.
00:37:36.000You know, you can go and you can try to look at how well did this content perform relative to other real authentic media targeting these communities.
00:37:46.000And what you see with the black community in particular is their Instagram game was really good.
00:37:53.000So on their Instagram accounts, the top five, three of them targeted the black community and got tens to hundreds of millions of engagements.
00:38:03.000So I would have to pull up the exact number.
00:38:16.000So it's amazing that they got the kind of engagement that they did, even without the sharing function.
00:38:22.000One of the things you can do is if you know the names of the accounts, and a lot of them are out there publicly now, You can actually see them in regram apps.
00:38:32.000So people were regramming the content.
00:38:35.000So Facebook says about 20 million people engaged with the Instagram content.
00:38:40.000But what isn't included in that is all of the regrams of the content that were shared by other accounts.
00:38:47.000So, the spread and the dispersion of this, it's an interesting thing to try to quantify.
00:38:55.000Because we have engagement data, but we don't know did it change hearts and minds.
00:39:00.000We don't know if it influenced people to go follow other accounts.
00:39:04.000We don't know if it influenced people to not vote.
00:39:07.000There's just so much more, I think, still to understand about how these operations work.
00:39:13.000We can assume that it had some impact, right?
00:39:16.000I mean, as you were saying earlier, when a new person enters into a conversation, it changes the tone of it.
00:39:22.000How much of what they did was their own original post, and how much of it was commenting on other people's posts?
00:39:29.000So, I thought you were actually going to ask a different thing there.
00:39:32.000Well, what did you think I was going to ask?
00:39:33.000How much of it was them repurposing our own posts, right?
00:39:43.000So, they created a lot of their own stuff, particularly in the early days.
00:39:47.000And so, you can actually read the data set.
00:39:49.000And one of the things, when we started finding these posts, I was struck by...
00:39:54.000How sometimes it read like ESL and then sometimes it read like perfect, flawless, professional English.
00:40:02.000And then other times it read like normal English, vernacular, just the way that we would talk to each other.
00:40:06.000And I started digging into what that was.
00:40:10.000So when it was vernacular English, when it read like fluent American English, it was usually cribbed from somewhere else.
00:40:18.000So they would go and they would find a local news story from some obscure local paper and they would crib and then they would paste that and then so the Facebook post would be that cribbed sentence from that article and then their meme.
00:40:30.000Maybe they would add a sentence underneath it to give it some kind of context or angle.
00:40:35.000When they would write their own stuff, you would see the sloppiness.
00:40:38.000That's where you could see subject-verb agreements not quite there.
00:40:42.000The ways in which Russian possessives are different than American possessives.
00:41:20.000And I'm like, it reads like, remember you're like in, you know, English in college or something and you've got to like write a formal essay.
00:41:28.000So nobody actually talks like this, especially not, you know, your stereotypical Texas secessionist.
00:41:34.000So it was funny seeing these incongruities.
00:41:37.000And that's unfortunately one of the best ways to tell what you're dealing with is actually to kind of look for those incongruities now and see as you read communications online, like, Does this read like an American?
00:41:57.000And what we started to see was one way to not get caught for your lousy English or your cultural lack of kind of Native language Abilities is to just repurpose other people's stuff.
00:42:16.000And so that's where you would see memes getting shared from on both the right and the left.
00:42:22.000You know, you'd see a lot of these like Turning Point USA memes that they were repurposing and pushing out.
00:42:26.000Or you would see Occupy Democrats or the other 98%.
00:42:30.000So memes from real American pages, real American culture.
00:42:35.000And they would just sometimes slap a new logo on and just repost it as if it was theirs.
00:42:40.000So it does in those instances read just like, you know, authentic American content.
00:42:45.000And in many ways it is authentic American content.
00:42:47.000How many people are working for this agency?
00:43:03.000They moved offices, and then people started calling in bomb threats to the office.
00:43:10.000And it was just like every day a new bomb threat would get called in, so they couldn't work, basically.
00:43:14.000I assume this is like some American intelligence agency just fucking with them.
00:43:18.000So there's people calling in these bomb threats to try to keep them from working.
00:43:21.000And I think there was an article that came out really recently that said that Army Cyber, one of our agencies, worked to just take them offline during the midterms, a couple days around the midterms.
00:43:35.000I wonder if whoever's calling it in is doing it in bad Russian.
00:43:51.000I mean, it only makes sense that in this bizarre and unpredictable and really unprecedented environment that we find ourselves in, that something like this would come up and just sort of throw a monkey wrench into the gears of real conversation online.
00:44:09.000It's a really amazing time in that we're getting to see this kind of stuff happen in real time.
00:44:16.000We're getting to see these sort of weird attempts at manipulating things.
00:44:22.000And I think in a lot of ways successful, especially with less sophisticated people that don't really understand that they're being trolled and that someone is fucking with them.
00:44:32.000And there's, it seems, I mean, I've, there's a bunch of accounts that I have bookmarked that I follow, but I don't follow.
00:44:39.000So I don't follow them online because I don't want them to know I'm following them, but I just go to them.
00:45:13.000I was looking at the conversation around GMOs.
00:45:18.000And because we have seen one of the things that Russia does besides the social bots and the, you know, the, you know, screwing with like Americans directly is the House.
00:45:30.000So this was a Republican House Committee, House Science and Technology Committee about a year ago said that they were Seeing evidence of both kind of overt propaganda and then ways of disseminating the propaganda.
00:45:44.000So there's always the dissemination and then the accounts and then the content.
00:45:47.000So it's like you look at three different things to try to get a handle on whether or not this is real or fake.
00:45:53.000So when we talk about the accounts, we're looking at are they real people?
00:45:59.000When you're looking at the content, you're usually looking at the domains.
00:46:03.000And that's kind of the last piece because you don't want to have any kind of bias get in there, but you're just trying to see is it being pushed through like overt Russian propaganda domains?
00:46:14.000And then the third is the dissemination pattern.
00:46:16.000Is it being pushed out through automated accounts?
00:46:18.000Is it spreading in ways that look anomalous versus how normal information would spread?
00:46:24.000So one of the things that the House Committee looked at was using that kind of rubric, Russian, you know, these dubious pieces of content and narratives around American strategic industries.
00:46:38.000So the energy industry, oil and fracking, for example.
00:46:41.000Or you see a lot of stuff with GMOs and agriculture.
00:46:48.000This narrative of Putin and Russia being the land of organic plenty in the United States, serving its people toxic, poisoned vegetables, this sort of stuff.
00:46:58.000And Meanwhile, at the same time, there's competition for who's going to get the, you know, large contract to provide rice to some part of the world.
00:47:08.000So there's like an economic motivation underlying this kind of narrative.
00:47:12.000And I was looking at one of these accounts, and it was tweeting an article about Hillary Clinton.
00:47:19.000A vote for Hillary is a vote for Monsanto.
00:47:21.000But it was tweeting this just like three months ago or something.
00:47:25.000It was like mid-2018 or late 2018 when I was looking at this.
00:47:29.000I'm like, well, that ship sailed a long time ago, guys.
00:47:32.000Why are we tweeting about Hillary's votes?
00:47:57.000So I would say not much, to be honest.
00:48:02.000There were 900, maybe 800, I think, tweets about vaccines in the content.
00:48:11.000And so Facebook and Twitter, you have this, sorry, Facebook and Instagram, you have this building up of tribes.
00:48:19.000Twitter, you have instead, they're just talking about whatever's popular, right?
00:48:25.000They're shitposting, they're talking about whatever's current and new, whatever scandal has just broken anywhere, you know.
00:48:31.000So, Twitter is less about establishing relationships and more about joining the conversation and nudging it.
00:48:38.000And so, most of the vaccine-related posts, it was not a big theme for them.
00:48:42.000It wasn't something that was on, like, Facebook and Instagram, where that's where they're really leaning in, like, this is what we want Americans to think about.
00:48:48.000So, no mention of vaccines on those platforms, not on YouTube.
00:48:51.000On Twitter, you see it in 2015, funny enough, during the Disneyland measles outbreak.
00:48:56.000Much like there's a whole lot of conversation around vaccines right now because of the outbreaks in Washington and New York, back in 2015 you saw the same thing.
00:49:04.000Lots of conversations about measles because the Disneyland thing that happened down here.
00:49:08.000And so they're in there and they're saying, Vaccinate your kids, don't vaccinate your kids.
00:49:13.000They had a couple of conspiracy theorist accounts.
00:50:09.000That's where, that's I think the thing that, you know, even with whatever, you know, political proclivities you may have, I think you can at least recognize humor, even if it's laughing at your side.
00:50:23.000And I will say that some of the stuff, especially targeting the right-wing, you know, the right-wing like youth kind of pages, they were funny.
00:51:23.000Be ashamed if anything happened to it.
00:51:25.000And that's the meme that they're putting out there when they're complaining that their fake page got taken down.
00:51:30.000There was tons and tons of these memes also about the Russians did it.
00:51:37.000Mocking the idea that the Russians did it.
00:51:40.000As the story is beginning to come out, before we've had the tech hearings, before we've had the Mueller indictments, before we've had the investigation, you see these memes where it's like, oh, my speedometer was broken, it must have been the Russians, or a picture of Hillary Clinton,
00:51:57.000and it's in a little golden book kind of thing, and it's like the whiny child's guide to blaming Russia for your failures.
00:52:38.000It's just, it's going to target you with the thing that you're most likely to be receptive to just because of psychological bias and tribal affiliation.
00:52:47.000And you're not sitting there thinking, how is this person who is purportedly just like me screwing with me?
00:52:54.000And that's why it does manage to attract a following and get retweeted, get reshared.
00:53:26.000We have a four-year-old's understanding.
00:53:30.000If you just grabbed a person, a random person on the street, a college-educated person, and asked them to describe what's so bad about Russia, wow, it's like a communist over there or something.
00:54:34.000Tell me where you keep the beef jerky.
00:54:37.000But that was in one of the, I think the Mueller indictment from February 2018. There have been three kind of big documents that have come out, two from Eastern District and one from Mueller on how it all worked out.
00:54:52.000I think another misconception is this notion of $100,000 in ads.
00:54:57.000They spent 18 million dollars in 2017, I believe, was the stat that came out during another one of the Mueller indictments.
00:55:04.000So they're not just, you know, the money is not just going for the salaries and the ad buys, the money is also going for, they were talking about using kind of consultants.
00:55:14.000And this is where you get at this thing that comes out during the stand-up where they're like, black people who are LGBT don't want to see white LGBT memes.
00:55:21.000And And this degree of granularity, the degree of sophistication, but then also what you see them doing is engaging one-on-one.
00:55:29.000And that's where it crosses the line from social media operation to this is much more like spying.
00:55:44.000But what's interesting is the—it does paint a pretty interesting picture of this couple under deep cover that are engaging with and pretending to be Americans and forming relationships with people.
00:55:56.000And apparently it's based loosely on real people.
00:56:00.000But what you see in the Mueller indictment is the text messages, the Facebook Messenger messages, where they're going back and forth with real activists— And they're saying things like, you know, hey, my ad account got shut down.
00:56:31.000We can give you some money for posters.
00:56:33.000And they're sending money for posters.
00:56:35.000Or they reach out to a Trump supporter and they say, like, we think it'd be really funny to have a Hillary Clinton impersonator sitting in a truck, flatbed truck, that's made up to look like a jail.
00:56:46.000Let us, you know, if we give you some money, will you find the Hillary Clinton impersonator and put her in jail and do this Hillary for prison thing?
00:56:54.000And so this is where another thing that they did was using Facebook events to create real-world protests.
00:57:02.000So they're not limiting it to shitposting online and making people feel tension online.
00:57:07.000They're actually sending people out into the real world to have in-street violence.
00:57:12.000And so one of the things that they did was they coordinated a Facebook event, one for the Texas secessionist page and one for the, there was a pro-Muslim called United Muslims.
00:57:23.000And on the same day, at the same time, in Texas, they have a rally to support Texas culture and resist the Islamicization of Texas across the street from a rally to defend Muslim culture.
00:57:38.000And so they, like, there's literally no, you know, they just create these Facebook events on these pages, and then they promote them with ad dollars and other things.
00:57:47.000And you literally, if you go and you look at the Texas reporting from that day, I don't remember if it was dozens or hundreds, but a sufficient number of people Showed up that they had literally on opposite sides of barricades, police officers in the center screaming at each other because one group is there for the resist the Islamicization of Texas and the other group is there to like defend Muslim culture.
00:58:09.000So you get two, you know, two opposite sides of the spectrum in the same place at the same time and you literally incite like a mini riot.
00:58:17.000So there were about 81 of these events where they were holding...
00:58:22.000Black Lives Matter-style rallies for victims of violence, police violence, memorials for people who were killed by police officers, you know, things that real Americans would do, but this wasn't being done by real Americans.
00:58:35.000And that's the insidious thing, right, is how does Facebook detect that?
00:58:41.000How do you, when you see this come to defend Texas culture and you're a diehard, proud Texan, you know, You're not thinking like somebody in St. Petersburg is organizing this.
00:58:54.000And that's the, I mean, I think that the idea that this was just some memes is just not, it doesn't respect the significance of what they were trying to do and how effective that they were.
00:59:11.000With these other things, or even if they're just trying it out just a little bit, just working to see what works, they're always experimenting, they're always trying to find ways to create that tension.
00:59:21.000And that's the thing that I think is so interesting about this, right?
00:59:25.000This evolving this idea of an information war where these tactics evolve, and you are really at a disadvantage when it comes to actually detecting them.
00:59:35.000Yeah, and on the outside, if you're looking at that, you'd say, well, okay, what is their objective?
00:59:39.000Why would they have this Texas secessionist rally across the street from a pro-Islam rally?
00:59:47.000You know, if you're on the outside, you think about the amount of effort that's involved in doing something like this, and they're also doing this with no leaders, right?
00:59:55.000There's no one there that's running it when they get there.
00:59:57.000So all the pro-Texas people go, here we are!
01:00:04.000I think a couple times there were comments on some of the like archived pages and things where you could see the screenshots of people being like, dude, you held us all come out there and like nobody showed up.
01:00:40.000They're not updating it, but I believe you can go and read it.
01:00:44.000And it was designed to call attention to police brutality type things.
01:00:48.000And so they have this Black Matters US page.
01:00:50.000And then there's the Black Matters Facebook page, the Twitter account, the Instagram page, the YouTube channel, the SoundCloud podcast, the Tumblr, the Facebook stickers.
01:01:01.000They had Facebook stickers that looked like little black panthers, like little cats.
01:02:00.000And the idea was that it was kind of vaguely militant-esque in that it was supposed to teach black people how to handle themselves at protests should there be police violence, how to fight back.
01:02:15.000And they actually went and found a guy, a physical fitness, you know, a martial arts guy, and they were paying him via PayPal.
01:02:22.000So he was running classes for the black community under this Black Fist brand, and they would, like, text him or call him.
01:02:33.000He played some of the voicemails on TV, actually, I heard them.
01:02:36.000After my report came out, I think they tracked him down, and he just talks about how they, yeah, they just PayPal'd him, you know, a couple hundred bucks every time he ran a fitness class.
01:03:06.000They hired these guys to be a fake YouTube channel.
01:03:10.000And it was called A Word of Truth, I think was the name of it.
01:03:15.000And so Williams and Calvin, these two guys, would give their word of truth.
01:03:20.000And their word of truth was usually about how fucked up America is, which, I mean, there are very real grievances underlying all of this, and that's the problem, right?
01:04:18.000So this guy who had been in the Williams and Calvin videos, so people recognized his face, in the 2018 midterms goes and says he wants to leave and he's going to leak all this information and...
01:05:11.000I know it's hard to explain without visuals.
01:05:14.000What they wound up doing was they did drop a bunch of docs.
01:05:17.000So they did release a pile of documents in which they claimed they actually hacked the Mueller investigation and Mueller had nothing.
01:05:25.000And so this is, again, another kind of convoluted piece of this where they do release information.
01:05:33.000And so in this particular example, they release information that we believe they actually got through legal discovery.
01:05:39.000So the documents that the investigation provided to one of the indicted Russians were the documents that they then leaked claiming they had hacked the Mueller investigation.
01:05:48.000So they're constantly doing these things to generate press, generate attention, create just that degree of people don't know what's real.
01:05:57.000Or they read the headlines that are then released by the more propagandist, overt Russian propaganda, and they think that that is the true story, that the Russians hacked the Mueller investigation.
01:06:35.000I do still regularly get surprised by the sheer kind of ballsyness and ingenuity of some of the stuff as it comes to light.
01:06:46.000Well, it's really fascinating that they went so far as to hire people to make a fake account on YouTube and hired these black guys to pretend that they're doing that on their own and they're really being hired by the Russians to And then when the guys leave,
01:07:03.000you don't know if they really did leave.
01:07:05.000You don't know if this is just more bullshit.
01:07:07.000It's like, like you were saying earlier, if they get you and you buy into it hook, line, and sinker, they win.
01:07:13.000If they get you to think, well, how much else is bullshit, they still win.
01:07:18.000Because you're looking at everything with sort of this tainted lens now.
01:07:23.000And in a sense, that's probably the ultimate goal, is to disrupt Our social media environment and to sort of hijack the natural conversations that are taking place.
01:07:37.000Yeah, and I think it's, I mean, it's effective.
01:07:39.000There's certain, you know, I was in Estonia last year, and they've been targeted by this stuff for decades now.
01:07:46.000You know, they have a 25% Russian-speaking population.
01:07:48.000Most of the news that they get is from Russian media, right, you know, right on the border.
01:07:53.000They talk a lot about the extreme commitment to educating their citizens, to make them realize that this kind of thing does happen.
01:08:11.000And I don't think we are quite there yet.
01:08:14.000I think that there's still plenty of people in the country who don't believe it happened or for some reason are completely incapable of separating the Russian social media influence campaign happened from it means Donald Trump's election is illegitimate or it means Donald Trump colluded.
01:09:27.000So that's where it plays out very differently depending on which part of the political spectrum you sit on.
01:09:35.000Well, it falls right into the issue that we have with cognitive dissonance.
01:09:40.000If we believe in someone or if we want someone to win, especially if it's our team or our person or on our side.
01:09:47.000You know, I saw a lot of this when Donna Brazile released her book detailing how the DNC sort of rigged the primaries Against Bernie Sanders and for Clinton.
01:10:00.000There were so many people that were Clinton supporters that just didn't want to believe it.
01:10:03.000I was like, well, why wouldn't you believe this woman?
01:10:05.000Like, you believed her before when she was supporting Clinton.
01:10:07.000And then when she leaves, now you won't believe her.
01:10:23.000It seems like they've been doing this for a long time, and they've gotten really sophisticated at it.
01:10:29.000And I think there's a lot of people that have been sucked into it that have no idea that it's actually influenced the way they've formed their own opinions.
01:10:38.000And they're so easily manipulated, many people are, that something like this, like a real good, solid, concentrated effort to try to target these groups that have these very specific interests and really dig in and form roots and then go out.
01:11:21.000And I'm always like, how do I properly convey recognition for the...
01:11:32.000I don't think we do ourselves any favors by pretending it all sucked and didn't matter and they're incompetent.
01:11:37.000I think that you have to acknowledge that you have a sophisticated adversary that is very capable, that is very determined, that is constantly evolving, and to treat that with the degree of respect it deserves.
01:11:47.000I think that that's just common sense, actually.
01:11:54.000I read media on both sides of the aisle and I, you know, I feel I try to stay current actually on what memes are percolating in lots of different spaces in part just because I am always curious about what's organic versus what seems to be disproportionately amplified or what new communities are popping up.
01:12:12.000I just think it's an, I think the spread of information among people is just a very interesting thing.
01:12:18.000It's something that interests me a lot.
01:12:19.000I think crowd psychology is really interesting.
01:12:22.000I think ways that crowd psychology has transformed as the internet has kind of come into being, particularly with things like the mass consolidation, the ease with which we can target people.
01:12:34.000We didn't even really talk about that, but the...
01:12:38.000Even in the decentralized internet, there's always been propaganda, there's always been crazy conspiracy theories, all this stuff.
01:12:45.000But it's that you can reach the people who are likely to be receptive to it now.
01:12:49.000And as people self-select into tribes, particularly in this country right now, one of the things that's remarkable is the way in which once you've...
01:12:58.000Self-selected into that tribe, and this is the media in your ecosystem, and you share it with your friends, and Facebook ensures that the people who see it are the people who are most likely to be receptive to it.
01:13:07.000Or if you run the ad targeting, you directly send it into the feeds of people most likely to be receptive to it.
01:13:14.000We have this interesting phenomenon where consolidation, targeting, and then these gameable algorithms mean that it's just...
01:13:23.000This kind of information goes way farther, way faster than it ever could have in the past, regardless of whether it's Russia pushing it or Iran, as we've seen a network of Iranian pages went down recently.
01:14:35.000In 2017 was when we started, like, we being independent researchers, I guess, people on the outside of the companies, academics, began to find the content, you know, really began to, investigative journalists would identify the name of a page,
01:14:52.000and then me and people like me would go and we would scour the internet looking for evidence of what was on that page.
01:14:57.000So I found a bunch of the stuff on Pinterest, for example, wrote about it.
01:15:01.000A guy by the name of Jonathan Albright found a CrowdTangle data cache.
01:15:06.000And with that, we got the names of a bunch more pages, a bunch more posts, and we had some really interesting stuff to work with.
01:15:11.000Originally, the platforms were very resistant to the idea that this had happened.
01:15:16.000And so as a result of that, they were...
01:15:23.000In, you know, there was a, the first thing that Zuck said in 2016, when, you know, Trump gets elected, Twitter, it goes crazy that night with people who work at Twitter, saying, Oh, my God, were we responsible for this, which is a very Silicon Valley thing to say.
01:15:40.000But what I think they meant by that was, their platform had been implicated as hosting Russian bots and fake news and And harassment mobs and a number of other things.
01:15:49.000And there was always the sense that it didn't have an impact and it didn't matter.
01:15:51.000And so this was the first time that they started to ask the question, did it matter?
01:15:59.000Fake news is a very small percentage of whatever on Facebook, the amount of information on Facebook.
01:16:04.000And the idea that it could have swung an election was ludicrous.
01:16:08.000So you have the platforms, kind of the leaders at the platforms, digging in and saying it's inconceivable that this, you know, could have happened.
01:16:19.000And as the research and the discovery begins to take place over the next nine months or so, you get to when the tech hearings happen.
01:16:30.000So I worked with a guy by the name of Tristan Harris.
01:16:34.000He's the one who introduced me to Sam...
01:16:37.000And he and I started going to D.C. with third fellow, Roger McNamee, and saying, hey, there's this body of evidence that's coming out here, and we need to have a hearing.
01:16:51.000We need to have Congress ask the tech companies to account for what happened.
01:16:56.000To tell the American people what happened.
01:16:59.000Because what we're seeing here as outside researchers, what investigative journalists are writing, the things that we're finding just don't line up with the statements that nothing happened and this was all no big deal.
01:17:11.000And so we start asking for these hearings.
01:17:15.000And actually, myself and a couple of others then began asking them, in the course of these hearings, can you get them to give you the data?
01:17:23.000Because the platforms hadn't given the data.
01:17:26.000So it was that lobbying by concerned citizens and journalists and researchers saying, we have to have some accountability here.
01:17:33.000We have to have the platforms account for what happened.
01:17:36.000They have to tell people, because this had become such a politically divisive issue, did it even happen?
01:17:42.000And we felt like having them actually sit there in front of Congress and account for it would be the first step towards moving forward in a way, but also towards changing the minds of the public and making them realize that what happened on social platforms matters.
01:18:03.000And it was really interesting to be part of that as it played out.
01:18:10.000Because one of the things that Senator Blumenthal, one of the senators did, was actually said, Facebook and Twitter have to notify people who engage with this content.
01:19:04.000I might have had a multiple day back and forth with some Russian troll.
01:19:09.000But that was, I think, one of the first steps towards saying, like, how do we make the platforms accountable?
01:19:14.000Because the idea that platforms should be accountable was not a thing that everybody agreed on in 2015 when we were having this conversation about ISIS. And that's where there's the through line here, which is, and it does connect into some of the speech issues too, which is,
01:19:30.000what kind of monitoring and moderation do you want the platforms to do?
01:19:43.000That we're really concerned that if we moderated ISIS trolls on Twitter, now not the beheading videos, there was sort of universal agreement that the beheading videos should come down.
01:19:55.000But if we took out what were called the ISIS fanboys, which were like 30-40,000 accounts at their peak, that we would, yeah, there's a document called the ISIS Twitter Census for anyone who wants to actually see the research done on understanding the Twitter network in 2015. There was a sense that,
01:20:11.000like, one man's terrorist was another man's freedom fighter.
01:20:14.000And if we took down ISIS fanboys, were we stifling their freedom of speech, freedom of expression, and like, goodness, what would come next?
01:20:22.000And that, when you look at that, that fundamental swing that has happened now in 2018, 2019, Where there's that same narrative because originally no moderation was taking place and then now there's a feeling that it's kind of swung too far in the other direction.
01:20:41.000But the original conversations were really...
01:20:45.000How do we make Twitter take responsibility for this?
01:20:49.000And legally, they aren't responsible for it, right?
01:20:54.000They are legally indemnified against the...
01:20:57.000They're not responsible for any of the content on their platforms.
01:21:01.000There's a law called Communications Decency Act Section 230, and that says that they're not responsible.
01:21:07.000They have the right to moderate, but not the obligation to moderate.
01:21:11.000Because they are indemnified from responsibility.
01:21:14.000So the question becomes, now that we know that these platforms are used for these kinds of harms and they are used for this kind of interference, where is that balance?
01:21:23.000What do we want them responsible for monitoring and moderating?
01:21:28.000And how do we recognize that that is occasionally going to lead to incorrect attributions, people losing accounts and things like that?
01:21:41.000Yeah, they're in a weird conundrum right now where they're trying to keep everything safe and they want to encourage people to communicate on the platform so they want to keep people from harassing folks.
01:21:53.000But because of that, they've also got these algorithms and they tend to miss Very often, like this whole learn to code fiasco, where people are getting banned for life for saying learn to code, which is about as preposterous as it gets.
01:22:10.000I think the learn to code fiasco is going to be the tipping point where a lot of people in the future, when they look back on when did the heavy-handedness become overreach, Learn to code.
01:22:22.000Because, I mean, Jesus Christ, I mean, if you can't say learn to code...
01:22:25.000I mean, I look at my mentions, I mean, on any given day, especially like yesterday, I had a vaccine proponent.
01:22:36.000It seemed like what was really disturbing to me was like the vast majority of the comments were about vaccines and so few about these unchecked diseases that are running rampant in poor communities, which was the most disturbing aspect of the conversation to me.
01:22:51.000That there's diseases that rob you of your intellectual capacity that are extremely common that as many as 10% of people in these poor neighborhoods have.
01:23:37.000But then there was the other accounts that kind of took it that step further and began to throw in like the ovens and the other stuff with Learn2Code, right?
01:23:46.000And that's one of the challenges with the platform, which is if you're trying to assess the...
01:25:05.000It sort of read more like a cell phone.
01:25:07.000Like, she had a picture of Facebook's logo in the image, and that violates the ad's terms of service.
01:25:15.000And the reason behind that is actually because Facebook doesn't want people putting up ads that have the Facebook logo in it, because that's how you scam people, right?
01:25:25.000And so, probably just like an automated...
01:25:28.000You know, an automated takedown, like an automated, like it halts the ad.
01:25:31.000You have to go and make some changes and then you can push the ad back out again.
01:25:34.000But it just happens at a time when there's like so little assumption of good faith and so little assumption of such extreme anger and polarization and, you know, assumption that the platforms are censoring with every little kind of moderation snafu that it makes it,
01:26:20.000And Vidja discussed that pretty much in depth when she was saying this is about moderating in scale when you're talking about millions and millions and millions of posts and a couple thousand people working for the organization.
01:26:35.000And then algorithms and computer learning that's trying to keep up and that's where things like learn to code and people are so outraged and pissed off because when they do get banned they feel like they've been targeted but you really just ran into some code and then it's really hard to get someone to pay attention to your appeal because there's not enough people that are looking at these appeals and there's probably millions of appeals every day.
01:27:01.000Yeah, and there's, you know, depending on which side you're on, you also hear, like, this person is harassing me and I'm demanding moderation and nobody's doing anything about it.
01:27:19.000It's interesting to look back at 2016 and wonder how much of where we are now is in part because not a whole lot happened in 2016. In 2015 in particular, very light, like almost no moderation,
01:27:34.000just kind of let it all hang out there.
01:27:39.000I look at it now, particularly as it evolves into this conversation about free speech, public squares, and what the new kind of infrastructure for speech, what rights we should expect on it.
01:28:00.000You know, I think some of it is almost like the people who hear the words free speech and they just assume that it's people asking for carte blanche right to harass and saying, you know, how do we balance that?
01:28:13.000I think Jack and Vijaya were saying this on your show.
01:28:17.000How do we maximize the number of people who are involved and make sure that all voices do get heard without being unnecessarily heavy-handed and moderating a thought or content and instead moderate behavior?
01:28:31.000And instead moderate particular types of signatures of things that are inauthentic or things that are coordinated and looking at this again gets to disinformation too rather than trying to police disinformation by looking at content really looking instead at actions and behavior and account authenticity and dissemination patterns because a lot of the worst trolls and stuff are just using these throwaway accounts and then they disappear.
01:28:57.000Well, I have the impression myself that when we're talking about censorship, we're talking about moderating content, that really we're talking about this current era and that what's coming is essentially we're like putting up a small twig fence and a herd of stampeding buffaloes on the way in terms of the more invasive or the more The
01:29:28.000more potent levels of technology that are on the way, I just feel like everything is moving in a very specific direction.
01:29:37.000And that very specific direction is less boundaries between people and information.
01:29:44.000And it's going to be insanely difficult or nearly impossible to moderate in 10 years.
01:29:52.000I just don't think it's going to be in the wheelhouse.
01:29:57.000I think we're entering into some weird place where we're either going to have to stay off of social media because it's just too toxic or grow a thick skin and just be able to deal with anything.
01:30:08.000And then if that's the case, how are we going to be able to differentiate between things that are particularly designed to manipulate us, specifically designed to manipulate us and change our opinions by foreign entities like this Russian troll farm?
01:30:29.000So if we believe that disinformation is in part facilitated by gameable algorithms, consolidation, and then the targeting, the kind of things we've talked about through this conversation, then I think that the algorithms piece,
01:30:45.000the manipulatable algorithms, that's really squarely the responsibility of the platforms.
01:30:49.000I don't think that there's any regulation or any kind of framework that's going to come from Congress that's going to address that.
01:30:57.000Well, that's pretty clear from the Facebook hearings, right?
01:30:59.000I mean, they barely understood the difference between an Android phone and an iPhone.
01:31:03.000They really don't know what's going on.
01:31:51.000As they change their algorithm a little bit, tweak it for the product function, which they just do in their role as business, there is no regulation that's going to come down fast enough to catch that.
01:32:01.000I think actually finance is an interesting parallel here.
01:32:05.000Because in the financial markets, there are these multi-tiered levels of regulation and oversight so that there's always some entity responsible, whether it's the exchange or a self-regulatory organization or the government and the SEC, looking to see if information integrity in the markets is being maintained.
01:32:23.000There's no shitty algorithm coming in to manipulate people.
01:32:27.000It's just making sure that we have that level of trust.
01:32:30.000So I think that right now the tech ecosystem is lacking regulation in all of its forms.
01:32:37.000But the argument for decentralization is, I don't know how you execute it.
01:32:42.000The antitrust thing in particular, as it comes up so much more now.
01:32:47.000Excuse me, I don't know under what economic grounds you make that claim that's way outside of my wheelhouse in my area, but there is something to be said for this, you know, return to decentralization in some way.
01:33:02.000I feel like it lets people have what they want.
01:33:06.000And it lets you, you know, Reddit's a great example.
01:33:08.000You have these, it's almost like federalism.
01:33:11.000You have this central platform, but then you have these little communities under it.
01:33:26.000This is the rules of the community, you're in the community, there you go.
01:33:29.000And this was how, in the olden days of the internet, like Usenets and things, you would have, this is the community that you've chosen to be a part of.
01:33:36.000If you don't like the moderation standards, you go to this other community.
01:33:39.000I think the concern with consolidation is that people who do get moderated away feel like there's nowhere for them to go.
01:33:48.000That they've lost access to the entire world.
01:33:53.000So I think if you have that decentralization, in some ways it It stops being quite so much of a freedom of speech issue if you can't speak on – if everything is like – if there's 50 different platforms and you fall foul of some sort of norms or standards or community membership in this one,
01:34:16.000you can go over here to this other one.
01:34:18.000Then the idea that somebody has moderated you away or deplatformed you or something is much less potent maybe.
01:34:26.000Well, on Reddit, though, if you dock someone or something along those lines, it has to be very egregious.
01:34:31.000Again, that federalism thing, you have the little moderation at the lower levels versus kind of top level, like you're summarily booted off.
01:34:40.000Yeah, that seems like the best approach, right?
01:34:42.000It seems like the best approach is to sort of let these communities sort of establish themselves.
01:34:47.000But even inside those communities, then you have people that gain power...
01:34:51.000Through moderation, and they start abusing it, and then it becomes some sort of a weird hierarchy.
01:34:59.000The decentralization in general is probably the right move for all this stuff, but how does that happen with something like Facebook or Twitter without government intervention?
01:35:11.000And, you know, that's one of the things that Tim Pool was bringing up.
01:35:14.000Like, if you guys don't do this, if you don't handle this, it's entirely possible somewhere down the line you're going to be regulated by the government.
01:35:30.000Well, you know, honestly, I say that and then I think back to the reality, which is in this Congress with this executive, I don't know how we get any regulation through.
01:35:43.000I think we've seen some examples, like the Honest Ads Act, which was introduced right before the first tech hearing, if I'm remembering the timeline correctly.
01:35:55.000So that would have been like Late 2017. And what they said was, you can no longer have a free-for-all with ads on social platforms where nobody knows who paid for it or where it's coming from or anything like that.
01:36:14.000So Senator Klobuchar and Senator Warner, and I think Senator McCain also was part of this, create this law saying that the platforms have to follow the rules that TV and radio already follow.
01:36:26.000And this is an example of recognizing the role that, you know, these are no longer startups that can't meet these obligations.
01:36:36.000It used to be that Facebook was exempt from these disclosure requirements because their ads used to be those, remember those tiny postage stamp-sized things that were on the right side of the page?
01:36:50.000They were given the same exemption that campaigns get for skywriting and postage and pencils, where it's like literally the form factor of the content makes it such that you can't put the ad disclaimer on there.
01:37:04.000And it used to be that all of the advertising on Facebook was regulated using that same finding that these postage stamp size things are too small to put the disclosures on.
01:37:15.000And then, of course, as we know, that evolved into the ads looking much like an organic post, and so now they do have these little things that pop up where you can see why you got targeted and what it is.
01:37:25.000I think that that's an example of the credible threat of regulation and the public opinion moving the platform to take an action that it wouldn't have necessarily done on its own.
01:38:07.000I feel like I'm probably not the best person to explain this because I don't know the specifics, but the GDPR was the law in the UK and in Europe that protects the data.
01:38:18.000It creates particular protections like you have to re-opt in for targeting.
01:38:24.000There are certain kinds of targeting that they can't do, certain types of data that you can request they delete.
01:38:28.000So this is a provision that took effect in Europe last year.
01:38:32.000We don't have that same law here in the U.S. We don't have the same data protections as the Europeans.
01:38:37.000And so California GDPR was the California state government, the state senate legislature, passing a law that basically mimicked a lot of the provisions of what the Europeans were given under GDPR. But what that did was it created a law that applied to the people of California.
01:38:55.000And so Facebook and Twitter and the others don't want to be in a position of having to have this, you know, kind of balkanization of legal requirements.
01:39:05.000And so they in turn have now, I believe, gone to Congress suggesting that we're going to need to have a federal solution that applies to all of the U.S., So a federal-level privacy regulation because they don't want to have to adhere to the privacy regulations of each individual U.S. state.
01:39:22.000One solution that's been tossed up was that people would have to somehow or another confirm their identity.
01:39:30.000That instead of it being an anonymous post, that it would have to say, Renee DiResta.
01:39:37.000You have to have a photograph, have some sort of a government ID. It shows that it's you.
01:39:42.000That we would somehow or another minimize trolling, minimize disinformation if your account was connected to a social security number or whatever it was.
01:39:53.000The problem, of course, is that these damn things get hacked all the time.
01:39:56.000And if your Twitter account gets hacked, now they have your social security number, they have your address, they have your information that you use to sign up.
01:40:06.000Unless there's some way, and there isn't, to absolutely lock down all that data and make sure that it's inaccessible to some sort of third party, that doesn't seem like a likely course of action either.
01:40:22.000The question of identity, I think most social science research that I've read has suggested that that's not necessarily the be-all and end-all.
01:40:32.000I think it depends on, per your point, what you're trying to do.
01:40:37.000I'm thinking right now of the—there is this request for— I think?
01:40:59.000You know those crappy data brokers, those horrible things where your name and your address is up there and no matter how hard you try, you can't get it down.
01:41:07.000So scraping those to grab names and addresses and then email addresses and leaving comments pretending to be those people.
01:42:09.000This was during some of the illegal immigration debates.
01:42:11.000Right as this was happening, people began complaining that immigration activists who were undocumented would not be able to run Facebook ads because they didn't have identification to verify with.
01:42:22.000So no matter what people put out, there's going to be somebody who has a complaint about it.
01:42:52.000For disinformation in particular, just to stay in my wheelhouse, we're trying to push towards multi-stakeholderism, which is just to say, can we create the back channels of communication that have come up for election integrity and things over the last few years,
01:43:13.000Can we create an oversight body, maybe the FTC, that is at least responsible for having some oversight to make sure the platforms are doing enough But this is, I think, this is going to be the theme of 2019. Does it go the antitrust route?
01:43:29.000Like, does it do a kind of hybrid combination of multiple, you know, tackling multiple problems at once?
01:43:37.000I'm really curious to see how we shake this out because it just seems like no, you know, even agreeing on what the problem is, we're not quite there yet.
01:43:47.000Yeah, and you are seeing calls, particularly from Elizabeth Warren, for breaking up a lot of these larger institutions, not just even social media, but even Amazon.
01:43:59.000She's talking about breaking up a lot of these bigger companies.
01:44:10.000And then, you know, what if one of those things that you broke up that becomes Twitter becomes more popular than the other Twitter, and it has much more attendance, then what do you do?
01:44:20.000Yeah, you should get one of those people on to just talk about that, because Lena Conner, or somebody who really, Matt Stoller, knows this space in and out, and I just don't.
01:45:06.000One thing that I notice is that my follower numbers doesn't move very much on Twitter as opposed to Instagram.
01:45:14.000I don't really use Facebook, but Instagram, there's a giant difference in how many followers I get per day on either platform.
01:45:22.000And it seems to me that the people that are using Twitter, they've kind of like locked in, they've found their little communities, and it's mostly toxic.
01:45:37.000It's probably not even 10% toxic, but it seems toxic.
01:45:41.000You know, when you look at those kind of comments and anytime something happens, it seems like the reaction to it is very rarely is it some sort of objective, rational discourse.
01:45:51.000It's most likely just insults and, you know, swears.
01:46:33.000There's no such thing in the history of America as a national public square.
01:46:38.000There are regional public squares or town squares, state squares, you know, where there is, again, this kind of federalism.
01:46:46.000People who have self-selected to live in a particular community, there's norms in that community.
01:46:50.000But if I were to go up to you in a public square and start screaming in your face or...
01:46:55.000You know, being an asshole and trying to get a whole mob together to go after you, like probably somebody would intervene, either a bystander or the police.
01:47:03.000And we have notions of like nuisance and things like that.
01:47:05.000We have notions of like, there's more of an intuitive sense of the balance between speech, which is to be protected, and then the kind of fighting words and that sort of thing.
01:47:20.000And when you're online, there is nobody who's going to come and step in and intervene in a way that would play out in real life.
01:47:27.000So I think that we just haven't quite ported those norms of basic good behavior in the real world into this I think there's actually a carryover to real life from social media that you can find in these protests at universities when conservative speakers come.
01:47:56.000And then Antifa wants to shut them down, and then you have people like the Proud Boys fight with Antifa.
01:48:32.000I mean, in 2000, in the 1990s, how often were there these incredibly volatile...
01:48:41.000Protests at universities where you have conservatives and liberals screaming at each other and you have these people that are being deplatformed and they won't let them speak at these colleges and then they're gathering up this online mob to try to bolster support and then people come to meet them.
01:48:59.000We're going to stop them at all costs.
01:49:01.000It's kind of flavoring real life versus real life being represented in social media.
01:49:10.000Yeah, I don't remember it from when I was in college either.
01:49:45.000Well, because I love the fact that for the first time in my life, it does not seem like the government has a fucking handle on how people are behaving and thinking at all.
01:49:58.000It's like you've got people that are...
01:50:01.000You know, people that are trying out socialist tropes and socialist ideas for the first time in the mainstream, and they're getting a big groundswell of support behind it.
01:50:11.000And you have a lot of people that are like pro-nationalists and pro-America for the first time in a long time, and that's getting a lot of support.
01:50:19.000There's more discourse now, even if it's toxic.
01:51:11.000I like the serendipity of the unexpected being pushed into my feed occasionally.
01:51:17.000Sometimes it's, you know, sometimes of course I get angry or, you know, feel annoyed or think, why the hell, you know, why this?
01:51:26.000But I think ultimately there is a lot of value to the platform.
01:51:31.000I think we're unfortunately, I really do believe that so much of the The polarization in the conversation around speech is people who got burned during the laissez-faire days of 2015-2016 in mentally linking up the idea of free speech with the idea of being harassed online.
01:51:51.000And I think when you look at—this is purely an opinion.
01:51:56.000I have absolutely no data to bear this out.
01:51:57.000But when you look at those Pew studies that show that younger people are more likely to want— Safe spaces or less offensive opinions.
01:52:12.000I do sometimes wonder if that's an effect of coming of age at a point when random assholes were screaming at you on social platforms 24-7 versus like, I didn't have that experience growing up.
01:52:24.000I didn't have that experience until I was like 25, you know?
01:52:27.000So, maybe there is something, per your point about that, there is not much of a difference.
01:52:39.000This is where you're having your social engagements.
01:52:42.000This is where you're having your conversations.
01:52:43.000So, it does shape the way people think about it.
01:52:48.000You know, their experience of what it means to have a conversation and what it means to speak freely.
01:52:53.000I think that's one of the interesting, you know, it doesn't help that free speech has sometimes, in many cases, become a fig leaf for, I want carte blanche to, you know, say all kinds of mean shit to people all day long with no consequences.
01:53:09.000Yeah, they think they should be able to do that because that falls under the blanket of free speech.
01:53:54.000All the things that make us human, those are all thrown out the window.
01:53:56.000You're just looking at text, and it might be coming from Russia.
01:54:01.000The text that you're getting upset at and responding to, I mean, per your research, really, there's a direct possibility it's not even a fucking person.
01:54:10.000Or a person, but not really representing their actual thoughts, just trying to push your buttons.
01:54:25.000We're in a unique time where it's hard to know who you're engaging with.
01:54:30.000It's hard to gauge whether it's good faith.
01:54:33.000I mean, I react sometimes where I'll see a response and go click into the person's feed to try to decide if I should take this as a seriously good faith inquiry or if it's like a kind of vaguely cloaked fuck you, you know?
01:55:42.000I've seen every now and then, if you follow reporters on Twitter, particularly ones who have open DMs, they'll periodically post the insane catfishing stuff that they get where it's all about money.
01:56:18.000I have tried over the years, whether it's conspiracy theorist communities or terrorists or Russia, Iran, the state-sponsored actors, the domestic ideologues, I have tried to always say,
01:56:36.000Kind of forensic analysis of this particular operation and then here is what we can maybe take from it and make changes.
01:56:45.000We've seen some of that begin to take shape and so I feel grateful to have had the opportunity to work towards connecting those dots and work towards having this conversation.
01:56:59.000Meaning helping people understand what's going on.
01:57:02.000I think I am not I am most concerned about the...
01:57:12.000As this gets increasingly easy to do, through things like chatbots, you know, now there's these, you've seen the website, thispersondoesnotexist.com?
01:57:25.000So there's a technology called, it's a machine learning technique, Generative Adversarial Networks, and basically they're, in this particular application, working to create pictures of people, faces of people.
01:58:17.000I think we haven't quite adapted to...
01:58:20.000What is it like to live in a world where so much of the internet is fake?
01:58:24.000And I do think, per your point about identity, that there will be groups of people that self-select into communities where identity is mandatory, you know, where this is who you are and you have some sort of verification versus people who choose to live in the world of, you know,
01:58:40.000drink from the fire hose, take it all in and try to filter it out yourself.
01:58:45.000So we look at these evolving technologies and I don't necessarily feel, you know, particularly optimistic in the short term.
01:58:55.000I think that ultimately it does, like we change as a society to a large extent in response to this.
01:59:01.000We think about, you know, there are going to be some fixes that the platforms are going to be able to undertake.
01:59:08.000They're going to be, we're going to get better at detecting this stuff.
01:59:11.000Maybe, you know, the adversary will evolve.
01:59:12.000Hopefully we get better at detecting it as it evolves.
01:59:15.000But I think we fundamentally ultimately change.
01:59:21.000People become more aware that this is a thing.
01:59:24.000That does change our ways of interacting with each other.
01:59:28.000But I feel like that is going to be the direction that this goes.
01:59:34.000The thing that keeps me up at night would be more the...
01:59:40.000The ease of turning this from a social media problem into a real-world war problem.
01:59:46.000Meaning, as an example, back in 2014, one of the first things the Internet Research Agency did, September 11, 2014, they created a hoax saying that ISIS had attacked a chemical plant down in Louisiana.
02:00:01.000It's called the Columbia Chemical Plant Hoax.
02:00:05.000There's a Wikipedia article about it now.
02:00:08.000But what happened was they created a collection of websites.
02:00:12.000They created fake CNN mockups, Twitter accounts, text messages that went to local people, radio station call-ins, you name it, everything, to create the impression that a chemical factory had just exploded in Louisiana and there was some attribution to ISIS. And this was done on September 11th.
02:00:30.000So this is the kind of thing where this actually did go viral.
02:00:33.000Like, I remember this happening not as a social media researcher.
02:00:35.000I just remember it actually being pushed into my social media feed.
02:00:39.000So you have these, and we didn't know that it was the Internet Research Agency for a year and a half after that.
02:00:45.000But this is the kind of thing where you look at parts of the world that aren't the US, like the recent drama between India and Pakistan, and you can see how these kinds of things can go horribly, horribly wrong if the wrong person is convinced that something has happened or if this leads to a riot or if this leads to real-world action.
02:01:07.000I think that's One of the main fears, as this gets better and better, the video fakes get better, the people fakes get better, what do you do then?
02:01:29.000I mean, we're going to get to a point where if someone's not in front of you talking, you're going to look at a video, you're not going to have any idea.
02:01:36.000You know, they're doing those deep fakes with famous actresses' faces and they put them in porn films.
02:02:11.000It's really stunning, and it's only going to get crazier and crazier.
02:02:14.000And it's going to be very difficult to, unless something is actually happening right in front of your face, it's going to be very difficult to be able to differentiate.
02:02:21.000And then I'm more worried about augmented reality and virtual reality and this stuff making its way into it.
02:02:27.000I mean, we're going to dive willingly with a big smile on our face into the Matrix.
02:03:56.000But this was just the first time where in the game you have a bat or something, and you're trying to beat zombies together with a bat, and they're all up in your face.
02:04:06.000I don't know if that thing ever came to market, but damn, was it good.
02:05:44.000It's very strange to think that those are not really people and that we are probably going to look at...
02:05:48.000All you have to do is create something that is so large and the propaganda is so terrifying that it causes you to act.
02:05:59.000Without double-checking, triple-checking, and making sure you verify the fact that something's really happened.
02:06:03.000And then it sets into motion some physical act in the real world that you can't pull back.
02:06:09.000And this is just not related necessarily, but what happened with Hawaii when they got that false warning that nuclear missiles were headed their way.
02:06:23.000And it was just someone hit the wrong button.
02:06:25.000I mean, if we come to some sort of a point in time where someone does something like that on purpose and shows you video that you really think New York City just got nuked and, you know, you have to head to the hills and there's a giant traffic jam on the highway and people start shooting each other.
02:06:45.000If Russia really wants to fuck with us, what they're doing now with just this IRA agency and all of these different trolls that they've got set up that's sort of trying to get people to be in conflict with each other,
02:07:01.000this is with primitive, crude text and memes.