Perla Ni, the founder of GreatNonprofits.org took a risk and organized a Stanford student group called Statistics for Good to figure out if their platform was suffering from a courtesy bias. Sometimes we don’t get the complete truth when we ask people to give us feedback especially when we know we are being judged or if we think we might lose a free service in the future. GreatNonprofits.org aggregates real feedback from the people that nonprofits seek to help, the question is are these people just being nice when leaving reviews and exhibiting a courtesy bias?
Resources
- Register your nonprofit with GreatNonprofits.org
- Statistics for Good Stanford group
- Datakind.org – submit a data analysis project you need help with
- Create unbiased online surveys
Musical thanks
Josh Woodward’s song “Tik Tik” was featured in this podcast. Find his ‘pay what you want’ music site: http://www.joshwoodward.com/
Transcription
Episode 17
Speaker 1: This is “Using The Whole Whale,” a podcast that brings you stories of data and technology in the non-profit world. This is George Weiner, your host and the Chief Whaler of wholewhale.com, thank you for joining us.
So I went to the doctor the other week and just got a check up. Don’t worry, everything is fine. They went through the normal things of testing this and that, and he went through the you know, “Do you smoke?” I was like, “Absolutely not.” And he went through the, “Do you drink? How many drinks a week do you have?” And that’s where you do the like, the quick pause, do the mental math and then just like, take 30% off that number and give them that number. And so what happens there actually happens to a lot of us when we survey our audience and we try to find the true answers and get that honest feedback. And it’s a good thing, I have to say, that doctors are not in charge of doing the average census for how much Americans drink. Because I think it would be woefully under reported.
Welcome to episode 17 of Using The Whole Whale. We are talking with Perla, the founder and CEO of greatnonprofits.org. A Great organization that allows people to leave reviews of non-profits they interact with. And we also have with us on this cast, Xioao, a current student at Stanford and she runs and is part of the Stats for Good program there that uses stats to help non-profits understand their data. So we are going to be diving into an interesting bit of work they did, kind of related to what happens when you’re dealing with the observer effect. Often referred to as the Hawthorne effect. Where in the act of observation, right as an observer, impacts the person who you’re observing. And so this happens, you know, on the surveys like we mentioned, that you’re maybe trying to get honest feedback for.
One of the things and effects that this can cause is what is called a Courtesy Bias. Wherein the person that you are potentially offering a free service to may not say that this was terrible. They don’t want to be mean to someone that just gave them something for free. And it was something that was kind of weighing on Perla’s mind at greatnonprofits.org. Where they’re trying to aggregate true feedback for non-profits from their beneficiaries. And I think they came up with an interesting solution of analyzing the data to figure out if in fact this effect was occurring. Let’s jump into the interview and see what they found.{2:51}
All right, we’re here with Great Non-profit’s, Perla and a wonderful person from Stanford who worked on some data analysis, Xioao. Hopefully didn’t mess up your name too bad there, Xioao. Could you guys introduce yourselves? Perla, can you maybe start here and tell us…what the heck? Right? Great Non-Profits sounds great? What do you guys do? {3:25}
Speaker 2: We’re a non-profit that helps donors and supporters find worthy non-profits and we also try to help give voice to beneficiaries so that beneficiary feedback can help non-profits and crew.
Speaker 1: Gotcha, what’s a beneficiary?{3:40}
Speaker 2: A beneficiary is anyone who benefits from a program of a non-profit. So that person sometimes is called a client. So a person who uses a homeless shelter or a patient served by a health clinic or a child that receives mentoring from a volunteer.
Speaker 1: Gotcha, and Xioao how did you come into this project?{4:01}
Speaker 3: I am a Master’s student in the department at Stanford and I interned at Great Non-Profits last summer and then Perla… Well, because she knows a bit about my background in stats, and like my interest in Great Non-Profits. So she approached me with this project with a focus on courtesy bias. And well, interestingly we have a very wonderful group named Stats for Good group at Stanford. So my colleagues and I headed out to take a look at this data set and carried out some analysis on this other set of data to see whether the courtesy bias indeed exists or not.
Speaker 1: Gotcha, Stats for Good, that’s a great name. But I gotta hold on here, what the heck is a courtesy bias and what does that have to do with Great Non-Profits and beneficiaries? {4:55}
Speaker 3: Yeah, I think…Perla can you address this?
Speaker 2: Yeah, so you know, in the business world collecting business feedback has been kind of like norms of the day. You know, whether you call it Amazon Reviews or it’s Trip Adviser Review’s or it’s the Net Promoter Score. Companies use these types of data to improve their products or services and strengthen their relationships with their customers. In the social sector we see it’s important to collect this feedback from clients. But still there is some critics who doubt the validity of collecting feedback from clients served by non-profits. Particularly you know, because the clients served by non-profits may not be very well educated, may be low income, may be different from the kinds of people that we are used to reading, you know, consumer reviews about.
And then the other concern was you know, these people because they’re getting these free services maybe they are, you know, more likely to just give positive responses. Because these are people getting free programs, access to programs, and feel grateful to the non-profits. So the courtesy bias is just a term in psychology used to… It’s a hypothesis about: Are people going to give credible responses to a question if they want to impress or somehow provide a positive feedback to the organization asking them for the information.
Speaker 1: Wow, so this could be detrimental actually, if it turned out that like the entire premise of Great Non-Profits was that it was a simple courtesy bias that was going on. So like, for example, just to play this out… If I were to ask you right now, “What do you think of my podcast? Isn’t it wonderful?” And hopefully you’d respond “Absolutely, it’s amazing!”
Speaker 2: Yeah, yeah.
Speaker 1: So that could really… I could see it become a huge problem for honest feedback, the transparency that you’re trying to provide in a not for profit sector. So I guess…how do you prove something like this? What was the process by which you went about approaching this problem?{7:04}
Speaker 2: So Xioao and the Stats for Good group at Stanford, they analyzed reviews from our database of reviews of non-profits that we’ve been collecting for the past 7 years. And they wanted to learn if in general, if in particular low income clients of food banks and homeless shelters were more apt to give positive feedback because they were so reliant on these free benefits provided by these non-profits.
Speaker 1: Gotcha, how big was this database, out of curiosity?{7:34}
Speaker 3: We extract about 2 million reviews from Great Non-Profits database. So all of them provided by different user groups. There are clients, there are board members and also volunteers and people who identified themselves as people from the general public. So we did some statistical analysis. Basically we looked at with the non-profit, where does the root score and the reviews provided by clients differ from the other groups. That’s basically the method we are using here.
Speaker 1: So for example, if you found that board members differed wildly from beneficiaries and it turned out in this case that beneficiaries scored much nicer we’ll say, in their ratings. That would be a sign that there’s a courtesy bias, correct?
Speaker 2: Mhm yes.
Speaker 1: interesting and so…the findings turned out to be that there was, there wasn’t? Like, how did you prove one way or the other, what was the result?{8:30}
Speaker 3: Yes, actually we did get some very interesting results. Great Non-Profits we got- we have the overall rating which is the average rating offered by all the users. If we take that as a fair score for non-profits then we will see that actually all scores provided by board members are actually biased. Because they are statistically higher than the average. But for the client’s consistent scored even slightly lower than the average. Meaning that they are not that positively biased to the non-profit at all, as people have assumed
Speaker 1: Wow so you may have even proven a negative bias here.
Speaker 3: Yeah…
Speaker 1: Where they’re actually more critical, they’re like, “You know what, that free food could have been a little better.” So Perla, out of curiosity here, what…what did you find as you were going through and maybe doing some of this analysis of reviews? What were beneficiaries most critical about when they were giving feedback? And this is maybe more a general question of your reviews.{9:32}
Speaker 2: It’s a very human touch. It’s about the friendliness of the person who is helping you. It’s often about how you were treated. Where you respectfully treated? The one analogy I make, it’s actually kind of like the hospitality industry. Where so much of your experience is dependant upon, you know, did you feel respected? Did you feel like you know, they listened to you? Did you feel like the person was friendly and helpful? So a lot of the reviews are I think about, that kind of human interaction and the quality of that human interaction.
Speaker 3: What you just said, you know, is absolutely true. And we also observed that when clients offered feedback, a lot of them are based…are about the details of the services. Like when then will talk about it, when they would like the services to be provided, the current schedule actually does not work for them and also they will offer very… a lot of very constructive feedback about the services. Like we would like this to be done this way…and then that…and also yeah. And also how their interactions between the staff have been. And also since like, yeah, a lot of ways in which the program can be like, improved basically.
Speaker 1: What are people less critical about in general that surprises you?{11:15}
Speaker 2: I think actually, things differ a lot depending on the types of non-profits. So for non-profits that offer educational programs or art programs. All the feedbacks we get are like, positive. In general positive, like I think people are just less critical to them. Or simply put, they are just awesome. Maybe. Who knows? But for programs like food banks and also non-profits that offer shelters for homeless people, actually the clients of those non-profits are more critical than other groups.
Speaker 1: Yeah, it’s interesting. It strikes me that you know, if you were more forward facing. You just mentioned like the service industry and hospitality industry. If you’re more forward facing as a non-profit, it seems like inevitably you’re going to get predominantly scores judged on that part of your business but not necessarily somebody poking through your financials and your you know, returns on impact, let’s say{12:15}
Speaker 2: Yeah, I would say that in general, those people like these reviews, that they are clients, they don’t have any visibility into the financials, or you know, they don’t know how to critique a leadership, that they…you know the Mackenzie kind of categories for organizational excellence are not the same ones that a typical client would use to form an opinion about a non-profit. It really comes down to kind of the day-to-day practical realities of the programs. You know, schedules, the people that are in the program office and the quality of their interactions, their friendliness to the person being served. Those seem to be much more visible to the client and the most important to the terms of their direct experience of the organization.
Speaker 1: Yeah, which is fantastic too because there are the Mackenzie type structures out there. The things that automatically come through 9-90’s. But this is giving you a window into something, a bunch of data actually, that you otherwise would not have access to. I’m curious, are there any examples of how this level of transparency has actually helped in the fundraising efforts or storytelling efforts of a non-profit?{13:30}
Speaker 2: Absolutely. Yeah, we have non-profits that 10-15% of their volunteers have read reviews on the Great Non-Profit site and then gone to sign up to become a volunteer for those organizations. We have a number of non-profits that told us that they have been able to raise more money as a result of being able to demonstrate from these stories. And even the organizations that received constructive feedback. They’ve come back and told us, you know, we actually have used the feedback. And so one organization, Communities in Schools, of [???] County, some of their students remarked that they would like to have a student lounge. And so they’re like, that’s a really great idea, that’s not too hard. You know, a couple of sofas and a coffee table and you know, they were able to put together a student lounge. And they felt they were able to improve the program and honor kind of the opinions and the input from the students they were serving.
Speaker 1: One of the things, as I was reading through their report that ultimately makes Great Non-Profits a unique tool that I think more non-profits should be using is, you avoided the courtesy bias also by being a third party. Because the courtesy bias relies on the fact that I just gave you, you know, a free bit of food, right? And then you would like more of that service. But if, you’re using a third party, you kind of deflect that to Great Non-Profits and you can be more honest and gather that type of a resource. And so I think that’s pretty fantastic. Is there practical ways that you’ve seen non-profits really leverage your platform?{15:30}
Speaker 2: Yeah so kind of the practical ways are… We really encourage non-profits to invite reviews from their community as beneficiaries. So the non-profits have done really creative things including, you know, they send out the emails, where you can write an honest and candid feedback about you, us. We’ve seen non-profits put up posters in their soup pantries saying, hey if you want to give feedback, you know, here is the website where you can give feedback about us. We’ve had even one food pantry, North Hope Community Outreach in Pittsburgh, Pennsylvania that put little slips of paper in the grocery bags they were handing out with the food saying, hey let us know how we’re doing. Provide feedback at this website greatnonprofit.org.
Speaker 1: The fear, I imagine, is out there, that what if somebody writes a bad review. What happens? How do you answer that? How do you approach that problem?{16:27}
Speaker 2: So the world around us, there’s feedback everywhere. And I think more and more so people are used to you know just, “Give feedback to sellers on eBay” or “Give feedback on a restaurant on Open Table after you’ve eaten there.” I’m not sure that there’s any industry that are immune to these trends in collecting data so that we can improve services. And we all know none of these sources of data are perfect. No one’s going to get consistently 5 star reviews, none of us are perfect. More and more so, I think people are used to triaging all this information… They seem like rational reasonable people, other reviewers seem slightly crazy and not like me. So I’m going to disregard that review. And so people are, people are used to triaging I think, you know, community sourced information. They know that… they don’t expect you to have perfect scores. They know there’s gonna be occasionally people who’ve had an odd experience with you or people whose reviews you don’t agree with. And that’s okay, as long as you know, in general we’re moving towards more transparency, more data, more feedback about; what are their services like from the perspective of a beneficiary and how can those services be improved? I think ultimately everyone in the non-profit sector wants to provide high quality services and programs. I don’t think anyone is out there just to like, be in the non-profit sector just to continue and maintain things exactly as it is. I think we’re all striving to achieve our organization’s mission.
Speaker 1: Well, you know, I think you did a very brave thing Perla, by turning to Xioao. And it’s a great, Stats for Good program. But basically you put it on the line and said, “Is this working?” And you addressed with data a potential underlying problem with the platform. So I, you know full applause for that. As we wrap up here, how do we….how do we find more non-profits for Great Non-Profits and then Xioao I’m going to ask you the same thing. So Perla, how do people leverage your platforms. In any non-profit, how do they do it?{18:35}
Speaker 2: You know the simplest thing to do is check out your Great Non-Profits profile. If you don’t have reviews yet you can just take that link to your profile, copy it, putting it in an email, sending it out to your beneficiaries. And asking them to give you some candid feedback on their experience in your program.
Speaker 1: Awesome. And then, Xioao, how do Non-Profits get in touch with the Stats for Good program at Stanford?{19:03}
Speaker 3: We have a website, so if you go just Google Stats for Good at Stanford and our website will pop up and they can just email me with their questions, their potential data set and we will be more than glad to take a look at that and answer whatever questions you have.
Speaker 1: Very cool, well how many non-profits does your group work with, a year, say?{19:21}
Speaker 3: Actually we are founded last year and so far we have been working with about roughly, I think around 10 and the number is growing.
Speaker 1: Right. Yeah I can imagine it will continue to grow. We hope to send you a few more as well. Well, I have to thank you both so much for your time and sharing your story here. Take care and we’ll be checking in on you in the future.
Speaker 2: Great talking with you George, thank you
Speaker 1: I really think it took a lot of guts for Perla to do that. To test that underlying assumption that people are leaving honest feedback for non-profits on greatnonprofits.org. And what’s more, now it even proves their model. That using this third party tool will actually get you unbiased feedback on how you’re really delivering services and what you can improve. So foundations I think should encourage the people they offer grants to, your non-profits should be doing this. There’s no reason why you shouldn’t be building up your reputation in greatnonprofits.org so people can find the true work as reported by the people you serve out there in the field.
That’s all I have for you. Though as always there’s going to be more research at wholewhale.com/podcast and in this podcast link we’re also going to give a link to a guide we created on avoiding biased question creation in your surveys. I know, fun stuff right? As always, really appreciate you listening. Thanks, take care.
This has been Using the Whole Whale. For more resources on today’s show please visit wholewhale.com/podcast and consider following us on Twitter @wholewhale. And thanks for joining us
Oh hey, I see you’re still listening, well done. Kudos to you. So last week Whole Whale released a new product called allgoodtext.com, I really encourage you if you’re interested in your data to sign up for it. What it does is it lets you pick out three key metrics from your Google Analytics account and it texts you every single week what happened in the past week versus two weeks ago. Super simple design to connect you a little bit closer with your data, make it a bit more actionable and make sure that you know that things are all good. Pun intended.
The music you are listening to now comes from Josh Woodward and you can find his music at joshwoodward.com. He’s been giving his music away for free, crazy right? He’s been doing that for the past decade and doing pretty well. Over six and a half million downloads and it’s a “pay what you want for my music” type of approach that is really a success story and really where the music industry ultimately is going. So I encourage you to check it out, talented guy and tremendous story behind it. Thanks as always and enjoy.
