This week we talk with Andy Aczel the CTO of the SpecialistsGuild.org about how they use quality assurance (QA) testing to change the lives of people on the Autism Spectrum. The Specialist Guild is able to train, place, and employ people on the spectrum while also doing great work to improve how websites and applications work for organizations.
Additional Resources
-
- Specialistguild.org Great usability testing firm
- Screenfly – screen grabs and usability across browsers (free)
- Browser Stack – test your site across devices (free tests)
- Selenium – automatic browser testing scripts
- Jira – mentioned in the cast used for bug tracking and project management
- FogBugz
– simple web bug tracking tool.
- Virtual Box by Oracle – Local virtual machine software that let’s you render full OS versions locally.
- Don’t Make Me Think – great book and resource for understanding usability testing.
Transcription
Episode 9
Speaker 1: This is Using the Whole Whale, a podcast that brings you stories about data and technology in the non-profit world. My name is George Weiner, your host and the chief whaler at wholewhale.com. Thanks for joining us.
Have you ever written an article and then gone back and tried to edit it? How did that work out? If you’re anything like me, you’re not your own best editor. And what holds true for writing certainly holds true for writing code and coding websites. Welcome to Episode 9 where we’re going to be talking about usability testing, but not just any usability testing. We’re talking with a very interesting organization called The Specialist Guild who are actually using usability testing not only to improve people’s websites but change people’s lives. They work with people who are on the high end of the Autism Spectrum, teach them how to do usability testing, and then help them get placed into jobs. Or they end up working directly with the Specialist Guild, helping improve the websites of other organizations. Let’s talk with the CTO, Andy, from the Specialist Guild and see how they’re doing.
Okay so we’re here with Andy, from the Specialist Guild, and Andy you’re the CTO over at the the Specialist Guild. But I’m wondering, tell us who you are and what do you guys do?
Speaker 2: Yes, I’m the CTO here. The Specialist Guild is a non-profit corporation that essentially was set up to create careers for people on the autism spectrum, particularly the high functioning people who don’t have a lot of services from government agencies. What we basically do is we take people who are not technically trained. We train them to do software testing at this point, we train them and [inaudible] to be foundation level standard and then we hire them as interns for some length of time, typically 9 to…9 months to a year. During that time, we take on contract work from industry and have those contracts executed by our interns. As a result they get both experience in doing real world jobs and also get a resume. Which is, what we found, a key to eventually being able to secure an open market position.
Speaker 1: That’s amazing. So you’re able to leverage technology to allow these folks to work remotely and what is the difference between just say, let’s train people and send them into the workforce versus what you’re doing? Why is this working so well?
Speaker 2: I think we are…right now we are basically unique in the industry, in the sense that there are other organizations that have used people with autism to do software testing, but they either are essentially a placement agency model. Which means that they may provide some training and then expect people to find their own jobs immediately after that. Or they are what I would call a workshop model where everybody basically works in one room. At the Specialist Guild, we have a slightly different approach. What we do is we give people you know, experience to be able to put together a resume and what we expect people eventually take on regular jobs within industry, just like everybody else. The ability to work remotely, where that comes in is that the… It allows us to be more scalable because we are not totally limited by how many people we can cram into a room. And as a result we can have interns working from their homes. And also this population occasionally has difficulties with the environment plus sensitivities, so being able to work from home, from a familiar setting, often allows them to build up enough confidence to eventually go out there and do the job.
Speaker 1: Love it, I love it. So, talk to me a little bit about top line numbers. How many people are you working with? Organizations that are your clients, essentially, what is your impact, as you measure it?
Speaker 2: Yeah well, you know, the impact is never as much as you would like. Money is always a problem when you’re a non-profit. And we are also… We have only been around about a little over a year and a half. And we have trained two cohorts of people. Half of those are currently employed in open market positions and the others are working for us as interns. They are approaching about a year, working as interns. So we are just starting our third cohort of 13. The starting is probably is going to be 10 people, 9 people, something like that. There is a little bit of uncertainty around it because we do the training in person. Although I have done training in the past over the internet. We found that that was a little bit less effective because it’s very difficult to…to… There’s no feedback from body language. So it’s very difficult to asses if people are lost and not following the material anymore. So I feel the training is better in person. Our training class is 10 weeks long and then after that people have the choice of coming to the office working as interns there or working from home by using essentially cloud based skills.
Speaker 1: Great, so it’s like each cohort has about 13 people?
Speaker 2: Typically we start with less than that, between 8 and 10. And the biggest class that I’ve handled happened to be 13. And typically the success rate is in the 60%… 60 to 70% range. So you know, the people who actually…well, one of the things about software testing is that you can be a very intelligent, smart ,individual and still not be able to comprehend -or don’t have the talent for perhaps is a better word- for this type of work. So the only real way to find out whether someone has the ability to do this work is to really put them into the training class and then over a period of 4 to 6 weeks it becomes clearly apparent whether both they enjoy this kind of work and are they able to do it? So typically we see a [inaudible] of about 30…35% and those people essentially select [inaudible] of the class. So the rest will complete the class, we are not always able to offer everyone an internship. Partially because it’s pretty dependent on how much work we can get. And you know, putting in a plug for that, I hope that more hi-tech people and companies are gonna be looking at companies like us, who do some of the best onshore rather than offshore to Asia.
Speaker 1: Yeah, testing for a good cause as well. And it looks like you guys go by the book here, I see some really great technologies here. You know you’re working with some very sophisticated…Jira, Selenium, can you talk a little bit about…first off, what the heck is usability testing and what are some of the tools you use to test?
Speaker 2: Yeah sure, well usability testing is just a label to basically say we going to evaluate the functionality of let’s say a website that someone has created from two perspectives. One is: Does it do what it was intended to do? And the other perspective is: Is the website actually usable by the average person who is going to be…the average constituent who is going to be going to that website. It takes a fair bit of energy sometimes to do that. But we find that people with autism actually have a shroud of characteristics or if you will, abilities that make them actually better at this type of work than what we call normal, typical people. Those are perseverance and the ability to be able focus on things for a long time. And they are able to find things that many of us, including myself, I can’t see them. As they are able to see these things. So when we talk about usability testing, we mean both on a functional level. Does it do what it’s supposed to do? For example, can you create an account? Can you interact with the website? But also, is it laid out in a way that is intuitively obvious for the average user.
Speaker 1: That’s fantastic. You’re using an actual strength of your stakeholders there to then create a system by which supports them. I absolutely love it. One of the big trends is certainly the mobile web and how the mobile web is cannibalising desktop. I know that you spend a good amount of time looking at how iPhone and iPad and Android (both tablets and phones) work. What are some of the most common issues that you see with a lot of, say, non-profit sites or even your client sites with regard to mobile?
Speaker 2: Yeah…I think mobile device use has created a lot of issues for really everyone who is on the internet. And the main problem with it is that there is a lot less standardization, lot more variants than there was in desktop, basically in the desktop world. You know, if we’re talking about the average consumer user, then they have Windows PC and they have Macintosh and that’s pretty much it. You know, here and there a Linux computer but typically not that much. So those systems were fairly standardized. You know…you didn’t see that kind of… As an example, you know there was maybe only 2 or 3 sizes of screens for laptops and etc. etc. What we find with mobile devices is that just about every device, even from the same manufacturer has sort of different quirks that have to do perhaps with screen resolution and the ability of how sensitive a touch pad is and other things of that nature that make it very difficult to extrapolate the behavior of the website from one device to the other. So a lot of companies are finding that it is actually quite difficult to ensure that the website is actually going to look reasonably good and be usable say, every Android phone in the universe. And as a result we actually do spend a lot of what we call multi-browser testing. Where we not only check several versions of browser theme-ologies, you know Chrome or whatever. But also browsers, say on a particular Samsung or LG phone versus IOS devices like iPads. So we are finding that these perhaps sites that are perfectly good on a PC environment are sometimes…you run into really faulty behavior on some of those devices.
Speaker 1: Yeah absolutely. And right now, I think around in 2013 the US average was about 15% on mobile traffic for overall internet usage and only I think 16% of US nonprofits have a mobile optimized site donation system. So there’s certainly a lot more work to be done, and should be being done. Can you give me an idea of what are the most common tools you guys are using over there to do this testing? Is it you know, and also like, cloud tools that you’re using? You mentioned Google Apps. What are some of the other tools that you use?
Speaker 2: So you know, typically when you interact with a customer, you have to have the ability to both manage the testing and then report the results. And in most cases they also interested in us directly interacting with the internal bug reporting system. So, Jira, you mentioned earlier, is for example, that is an open source bug reporting system. As a rule, what we do is we tell the customer, you tell us what system you use, we’ll learn how to use it. Because, you know, we want to make the interaction between us and customers as seamless as possible. Now, when they do testing for smaller or medium sized non-profits, of course this is a totally different story. But I’m talking about commercial clients and things. The… to return for a minute, to this multi-browser testing. It’s almost impossible for anybody to have all of the different devices at hand. Even really, all the different operating systems on a desktop setting. So what we typically do is we use our favorite off cloud based emulation technology. I’m not sure you’re familiar with it but an awesome example, there are cloud based sites where you can scheme up essentially a control machine of any sort. So you can say I want Mac OS 5 with you know, a particular version of the browser and you can do that. And you don’t actually have to own, you know, 15 different types computers with many different OS’s on them.
Speaker 1: Okay, just to jump in here. Andy is talking about using emulators online, and this just means using websites that have applications that pretend like they’re your phone, they’re your iPad, and can mimic what it’s like to use your site through different browsers and operating systems on different devices. So instead of running out there and running up the old credit card bill buying every single device, you can use some of these online emulators. Such as browserstack.com, and also screenfly.com to emulate these things. Although I have to offer a word of caution as you jump into testing every single browser and version on every single operating system. You want to actually look up what the market is really using, where it’s going, but also you can use Google Analytics in order to look at the top devices that your audience is actually using. So for example, if Internet Explorer 6 doesn’t render your site that well but then you look at your actual traffic and realize that only .05% (a.k.a maybe just your boss) is using IE6, you can make the case that it doesn’t make sense to make sure that your site is working perfectly in that particular browser. Because it represents such a small minority. We recommend setting a threshold, probably 10% of audience usage of a device in order to be supporting it fully. All right, let’s get back to the interview.
What’s currently stopping your organization from scaling and what size do you eventually want to get to?
Speaker 2: Yeah so, I think that’s an excellent question. I mean, what’s stopping us from scaling is resources. The…quite frankly, you know, there’s always financial challenges when you’re a non-profit. And also, right now I do all the training so I’m sort of a bottleneck in all fairness. But what we are hoping to do is, on our side, on the Specialist Guild side, is that we are hoping to go big enough to be able to hire some staff to expand the program, our area. But the other point I want to make is that we also are strong believers of the business model that we have created and we feel that other organizations in grown communities should create their own opportunities. Software testing isn’t going to work in every meetings I sit in, in America or small towns. So you know, what we are hoping is sort of, other organizations gonna look around in their own communities and see what careers people with autism can be engaged in. And create those opportunities basically following the same business model. Which is, create the appropriate training, provide an internship so that they can get confident and experience, and then have the local… I guess community, hire the people. Because they make fantastic employees, I can tell you that.
Speaker 1: Mmhhmm
Speaker 2: It’s always been a terrible idea for people who actually do the work to actually do the testing because of author bias. We all suffer from it, it’s got nothing to do with you know, a person being capable or not capable. It’s just doing exactly what the website is supposed to do, so you only test what it’s supposed to do and then… You know, just recently we had an incident where a very big client showed an interest in it, where it was full links on the stage and if you clicked, clicked them one of the times in the order that they were laid out, they all worked. But the minute you clicked them out of order, they took you to an error page. And…
Speaker 1: That’s amazing.
Speaker 2: …that office did not discover this and it took my guys two seconds to find it, so…
Speaker 1: Yeah that’s great, sometimes it’s the, you know, the curse of knowledge. The fact that you were already inside the system, you know how it should work and you can’t imagine anybody not understanding, it can be…
Speaker 2: Exactly. You got it, exactly right. So I’ve called… I’m a developer myself. I spent 30 years in hi-tech and I have done this. I have made mistakes like this. And you know when I saw, when they guy was telling me, “Hey look at this Andy,” I said “Okay, I think I’ve seen this before.”
Speaker 1: I guess, as a final… I know you work with like amazing organizations like Donors Choose, and I look forward to figuring out if there’s projects that we all can work with you on. As a final thing, how do people find you online?
Speaker 2: I wish they could find us easier, right now they… Obviously we have a website, specialistguild.org and I’m hoping that it’s fairly clear and understandable. And they can reach us through the website or they can reach me at [email protected].
Speaker 1: Great, thank you so much for jumping on the podcast today, take care.
If this isn’t a story about using the whole whale, then I think I have to pack up my things and swim home. What the Specialist Guild is doing, finding people that are on the autism spectrum and then training them to do usability testing is a perfect example of leveraging not only technology, but also then turning around saying how do we both improve the quality of websites while improving people’s lives. Again, there’s nothing not to love here. So think about the last piece of software you interacted with, maybe it’s at your organization or elsewhere and think to yourself if there were usability tests and software tests being done, couldn’t this be a bit better? And why not, if you’re going to be looking at improving these softwares, why not pull in the Specialist Guild to do what they’re tremendously good at and help you test your next site?
Well, that’s it for today. Thank you, again for joining us. Online resources, you’ll be able to find at wholewhale.com/podcast. Take care.
This has been Using the Whole Whale, the podcast. For more info on the topics covered in today’s show please check out wholewhale.com/podcast and consider following us on Twitter at @wholewhale, thanks for listening.
So I don’t know if I’m going to make this a habit or not but I see that you’re still listening so I’m gonna keep talking. As a follow-up to this episode, turns out that Whole Whale will be working with the Specialist Guild on a usability project for powerpoetry.org. Power Poetry again, is the largest online place for teens and poetry right now, and we’re super excited to see how that goes. All right, take care.
