Wrapping up our time at build 2019, Cory and Brett sit down with Mary Bellard, Senior Accessibility Architect for Microsoft. We hear how Mary and her team are using AI to help give people with disabilities equal access to data.  Mary also shares a few of Microsoft’s initiatives and products around accessibility; like the Seeing AI mobile app and the InnerVoice project as well. We also hear about the Microsoft AI for Accessibility grant, which helps ideas receive funding to become real products that can affect change with a global reach.
Did you know we are on Spotify?  Tune in to The Big Data Beard!
Music from this episode is by Andrew Belle.  Please go check him out…you’ll thank us!

Video available here:

Full Transcript:

– [Man] Welcome to the Big Data Beard Big Build conference 2019.

– Hi everybody this is Cory Minton with the Big Data Beard team and we are at Build Conference in Seattle Washington. And we are excited today to have another one of these conversations around how we use technology to make the world a better place. And we’re joined by Mary Bellard, Senior Accessibility Architect for Microsoft. Mary how are you doing today?

– I’m doing awesome thank you so much for taking the time to chat with me.

– Absolutely so Mary tell us a little bit about what you do for Microsoft.

– I run our innovation efforts around accessibility so accessibility for many people who may not know is about giving people with disabilities equal access to information through technology. So whether it’s a mobile phone or a laptop or some kind of wearable, people are accessing information and going through scenarios and experiences related to productivity and entertainment and people with disabilities want and deserve access to those same things. And so accessibility is all about finding ways to make all kinds of technology more useful and more inclusive for everyone. And so within the innovation space we are working on a variety of different things many of them AI powered, the project that I’m probably the most involved with is our Seeing AI mobile app for the blind and low vision community.

– [Man] Cool.

– But we’re trying to do more of those kinds of things so research projects and also products that are out in the market today that people can use.

– Very cool so give us a little bit of the background because I think people underestimate how many folks have disabilities in their lives and they have needs around accessibility talk to me about how big of a deal this is in terms of how many people were affected by these and how much this accessibility project can have an impact.

– The number of people with diagnosed disability in the world is over a billion. So that’s a huge demographic of people but then you add on top of that, you know if you unpack the scenario of disability experience it’s this idea that different kinds of input and different kinds of output are useful for someone. So maybe I have a broken arm and am using speech input for something. Or maybe I’m carrying groceries in both of my hands and I can’t use my hands to interact with my phone and would rely on my voice as an input. So when those types of situational limitations that benefit of accessible technology extends to beyond the billion plus into everyone that’s using any kind of technology.

– That’s cool, so when did Microsoft start to investing this accessibility initiative.

– Accessibility has been around for a very very long time Microsoft is not the only company that is doing it. Lots of tech companies are innovating in this space the Microsoft journey has been particularly important to me that’s why I moved across the country. Originally from Brooklyn, New York I’ve worked in accessibility for over 11 years and five of those years have been on the West Coast with Microsoft. And we have had a variety of different features within our Windows and our Office products. They continue to get better year over year and about four or five years ago we really changed the way that we were investing and the way that we were scaling the accessibility expertise across the company. And that was all led by Satya Nadella’s investment in inclusive design and also the leadership of Jenny Lay Flurrie who’s the Chief Accessibility Officer at Microsoft.

– So where does your passion for accessibility come from? ’cause it’s clear you spent some time in it but what kind of fuels your fire the fire in the belly to work on these accessibility projects?

– I am a natural problem solver. I am very into taking something that should be different and working to make it better. But also I genuinely feel and know that accessibility is the coolest kind of technology topic to be working on. It is something that offers real value to an individual but then you can also change the way, what I would call mainstream technology is also changed. And we see that a lot with AI, Seeing AI has impacted our computer vision capabilities. And so this power for the customer that then gets extended back into the platform is just really exciting.

– That’s cool so talk to me about there’s this macro project or I guess initiative within Microsoft called AI for good which is it’s more that just accessibility it’s kind of a larger project. Talk to us about what Microsoft’s AI For Good initiative is all about.

– Happy to. So like I mentioned we have all of these efforts around our own product portfolio Windows, Office, Azure, Power BI, all those great things and we have learned over the years that there is a lot of people outside the company who want to use our platforms to change and improve their own products. And so AI For Earth was the first AI For Good program. And it was focused around climate change and biodiversity and lots of Earth related topics. And we kind of took their model and wondered if it would work for the disability and the accessibility innovator community. And so we under the AI For Good umbrella we have announced our AI For Accessibility program we just celebrated our one year anniversary.

– [Group] Congratulations.

– There was cake.

– Always have to have cake.

– Yes, and we have $25 million over the next five years to empower anyone whether you work at a company or you’re an individual or you’re a researcher, to find value in our AI platform for your own research or your own product. And so we have three categories as the program exists today. Employment, the unemployment rate for people with disabilities is more than double that of people without disabilities and we want to have a positive impact in that. Daily life, so being able to deliver technology solutions that offer practical value to an individual so jumping back to Seeing AI going to a restaurant for someone who’s blind or low vision reading a menu may be problematic and a tool like Seeing AI can help read that menu in whatever way the individual wants. And communication and connection. ’cause underpinning so much of the topic of access to technology and experiences and opportunities is related to how we communicate. Being able to voice our concerns and being able to consume other people’s voices. And that’s a big factor. And one of our most recent ones, Inner Voice is working on that communication connection angle.

– Very cool, so why is artificial intelligence, it seems like it’s the basis for so much of this conversation and we talk a lot about AI and machine learn and big data but why is AI to you so important in the accessibility initiatives?

– So I mentioned earlier this situational limitation piece, the idea that at the fundamental level accessibility is about giving people however they choose to input or output access to that information. And AI offers very unique ways of collecting information and conveying that information to the end user. So being able to take speech and put it into text. Being able to take text and put it into speech. Being able to understand objects in a photo. all of these things are different ways that information is collected, processed and also really creative ways that we can then give that useful information back to the customer.

– Very cool, so you’ve talked about Seeing AI a couple of times.

– Yes.

– That’s an interesting project I had a chance to see Saqib give a demo with Rohan a couple of weeks ago at Spark, but not everybody’s seen it so explain that in a little bit more detail. ’cause that’s one of the banner projects that you guys have and it blew my mind when I saw it. So help the audience understand what is Seeing AI and how do they consume that?

– Totally, I’ll start with where it all began. So back in 2015 we had a hackathon, that’s a cross country initiative. So all of our employees are asked stop doing your day job and go do something else. And so the something else that myself and about 29 other people across the company went and did was Seeing AI and it is a mobile app on ios available for free and it is demonstrating the capabilities of our computer vision within Azure. We do that through nine different we call channels you could also call them features and so they’re divided up by the type of processing that they’re gonna do. So if a person has paper currency and they want to know which is a $20 bill and which is a $1 bill they would go into the currency channel, take a photo of that bill and then understand, they’d be told the denomination of it.

– [Man] That’s cool.

– If you are in a restaurant and you are trying to read that menu you could go into either the document channel or the sort text channel and it will digitize the previously inaccessible physical text on the document and these types of things are very practical for the customers but also are a great demonstration of the state of computer vision today. And we have all these channels that are what we call production ready and then we have some more experimental ones.

– [Man] R and D stuff.

– Not sure if it’s always gonna be right. And so handwriting is one of those. And in the handwriting channel a person can take a picture of any kind of handwritten text and digitally it gets analyzed and they can decipher what that message was. And we release that around Christmas time and so people in the blind and low vision community were reading Christmas cards for either the first time or the first time since they had lost their vision. So it’s a really cool thing. So it was released publicly in 2017 we continue to update it and it’s a really great demonstration of computer vision. It was buiLt with and by the blind and low vision community which is so fundamental to why it has the success that it has. And we’re excited where it’s gonna go.

– So you have to tell me some more stories–

– Yeah.

– People are using Seeing AI.

– Yeah.

– Because actually I saw Saqib was one of the folks that gave a great demonstration of it, walking up and it actually identified Rohan was 10 feet away. What are some other of your favorite stories around people who are using this to change their lives?

– Well the, I mentioned the Short Text channel and so if you have a full document like a menu there is a document channel that guides the user to detect the edges of the page to take a good photo and so that’s really great for a lot of scenarios but for scenarios where you just want to get real time on device processing we have the short text channel. And what we learned from social media is that we had customers who are watching foreign films with subtitles and they were mounting their phone up, pointing it at the screen and every time the subtitles changed it would read aloud what the subtitles were on the screen.

– Yeah for sure.

– Which was not at all how we designed it or ever thought that someone would use it but cool that people are finding value in it.

– Yeah that’s very cool. You talked about the Seeing AI I think is a good statement around the state of computer vision today.

– Correct.

– So what is the state of computer vision today? ’cause I know there’s a ton of research the number of scientific and research papers being published in this area, it’s stunning the volume of them. What’s your view on the state of computer vision today?

– I think it has made great progress in detecting objects, we are beginning to see value in the personalization of computer vision so what’s available in Seeing AI today is I can take a picture of a person and get some information about a guess on their gender, their age, their facial features et cetera, would likely say that you have a beard.

– That’s a brilliant identifier.

– Just a little one.

– But we can also on device we can store photos of an individual who’s like my friend and so I would know that this is Cory or that this is Brett and so that personalization is beginning. We’re also really curious about, we being society not necessarily just Microsoft, being able to detect personal objects. So how can I find my car keys in a space compared to just any car keys? And we need more data for that. So the state of computer vision is very useful and lots of room to grow especially in the personalization space.

– Did you see the team for the Imagine Cup finder and what they were doing around object detection with the camera and the phone? I thought that was really cool.

– Yeah the Imagine Cup has been a fantastic pipeline to be watching over the past several years. I mean there have been tons of accessibility related projects.

– It’s incredible they’re 18 years old.

– Yeah.

– It’s incredible, mind blowing.

– And what underlies some of that is accessibility has not really been discussed with the intention that it deserves at undergraduate and even high school levels up until about two, three years ago when we saw a really big rush with that. Because we hire fantastic talent into our companies and in their computer science classes they may not have learned about how to make that user interface accessible or how to make that website accessible. And we’re seeing that shift which is a really great thing that we’re able to contribute to. ’cause building accessible technology we know is fun and it’s great when we see young people taking on this opportunity because it makes the future for technology even brighter.

– What’s the right thing to do to be inclusive? So to teach that early I think is brilliant. So one of the things we obviously talked about Seeing AI but AI For Good has a bunch of other cool projects, you mentioned one this Inner Voice thing.

– Yes.

– And this one I want to dig into a little bit ’cause I saw a video and these kids, it was all I could do not to just tear up thinking about how cool this is. Tell us about what Inner Voice is and how that came about in the AI For Good group.

– Right so again, AI For Accessibility has been around for a year, our application window is open at any time and just about every quarter we announce the grantees that have been selected in that round. And so one of our recent rounds has been a company called iTherapy that works on speech and communication assistance for people who maybe have autism maybe they have had a stroke and they are helping an individual practice their communication skills. Their emotion, understanding of others. And just the communication value that they have with an app that is available today. They had this great product that had success in a variety of schools. And they were looking at Azure and wondering if that could bring their product to the next level. And so they applied for the grant, we awarded that and what they’re doing now is making their digital avatar that they have in their app powered by our facial recognition technology. And there’s also they’re working on a chat bot to help an individual, and in the video it’s children, to practice interacting with the avatar. Because this idea of mimicking others emotions is very fundamental to progressing your own understanding of emotion and your own ability to express that emotion.

– Very cool, so tell me a little bit about are there any other cool projects inside of AI For Accessibility that are available like now that you’re working on?

– Yeah we have another grantee Zyrobotics who has a product called ReadAble Storiez and that’s around reading comprehension and literacy. And they also have a very intentional intersection with people from low income neighborhoods. And they have used Azure in our Ai in our machine learning to influence the accuracy and the capabilities of that reading comprehension tool that is also available today.

– Very cool. So do you wanna go with this one?

– Yeah.

– Sorry, get ’em shifted over to something.

– So what announcements Build regarding accessibility are you most excited about?

– I think the speech capabilities within Azure are always going to have a lot of weight in the accessibility space. We have had a number of partnerships with the deaf and hard of hearing community over the years being able to make captioning, automatic speech recognition for captioning, Real is something we’re very proud of. We have a relationship with the Rochester Institute of Technology in upstate New York where they have a large population if students who are deaf or hard of hearing. And their in house captioning service which is real people typing captions was not able to scale to match their student population and so they asked Microsoft what kind of technology we had and what we worked on together was using our Microsoft translator app, which is also available for free. It offers speech to text it also offers language translation. Many classrooms at RIT are using this. And the kids who are deaf are finding value ’cause they’re getting access to the information from their instructors but also the kids who are not deaf are also benefiting from it. Maybe it helps them stay focused maybe it helps with note taking. So variety of really–

– Comprehension, I saw during of the presentations this week where it’s literally above the slides you get in that transcript. For me I’m one of those folks that when I read I comprehend far better than when I hear. And wile I am deaf in one ear but it’s like this is incredible, this is an incredible thing for not just people that have a diagnosis as you said accessibility challenge it’s like holy cow, it just made meet me where I am that I don’t classify as maybe a disability it’s just like this is how I work.

– Right.

– So I’m curious, Microsoft seems to put a lot of focus on creating avenues for creativity in a space. You talked about hackathons you talk about these grantees. How does that all come together? is that just the ethos of the company? Is that something that top down? Where does that room for creativity come from?

– We’re encouraged to take risks. There’s definitely a commentary on the culture that we have within Microsoft globally. And I think the other aspect within the accessibility space is understanding and partnering with the NGO community around the world. Yes of course we hire people with disabilities into Microsoft but there’s also a lot of disability related non profits around the world. And it’s important for us to have those relationships to understand what they want with technology, what they expect from technology. And so that’s a big factor in keeping it real we’re not just creating tech for tech’s sake. We’re creating technology that people want and that deliver a very valuable solution to them.

– That’s very cool so in all these creative places you’ve got opportunities to do this there’s gotta be like new, cool projects you’re thinking about. So what are the things you’re most excited about in context of AI For Accessibility. What are the cool projects that your team’s working on that you can speak about. But just things that you’re generally looking at over the next 12 to 18 months?

– I think because you can’t have any conversation about AI without talking about data I would say that data is a very very hot topic for our year two of the grant program. And that’s across AI For Good, it’s not just for AI For Accessibility. But within AI For Accessibility there is not enough data from people with disabilities in these data sets. Jumping back to the Seeing AI example for a moment, a person who is blind is gonna take a different framed photo of an object of any person who is sighted and so how do our models understand that? And for people with impacted speech. How does speech recognition work for them? So again with RIT we have a relationship around captioning but we also have a relationship with them around building custom speech models through Azure Custom Speech. And one of the professors is also deaf at RIT and he demo’d at Build last year on stage with his deaf voice or deaf accent and the speech recognition system was able to pick it up.

– [Man] That’s awesome.

– But that’s only possible because of the data that was used to train that model. And so we definitely have a great opportunity for talking with the disability community and helping them lead us to the solutions that they want. And helping us with the data collection to enable those possibilities.

– It’s interesting when you talk about that I see this massive intersection between AI can power accessibility but then there’s this other side of it which is data and the way that we build AI models it’s really hard not to immediately identify outlier things like things that are… So we had an ethics talk yesterday and it’s really hard not to teach them neural network that this that this is an obvious problem so how do you marry those two challenges of you’re trying to collect data that is about a specific community but you’re equally trying to protect and honor that community’s privacy how does Microsoft approach that and how do other organizations try to think about that?

– It’s an ongoing conversation that privacy comes up in all aspects of AI. I think the biggest thing that we can do is be transparent about what we’re trying to do. And if I think about sign language recognition with the deaf and hard of hearing community being able to collect a large corpus of video data of people signing, we want people from the deaf community to contribute to that. So we have to be very transparent about how we’re gonna collect it, how we’re gonna store it and luckily Azure has great things to offer in that space. But also be transparent with the community about what we intend to do with it. The fact that we could make sign language a translatable language in the way that spoken languages are is exciting from a technology perspective but is it something that the deaf community wants? So it has to be an ongoing conversation. And as technology gets more accurate then the conversation can change and hopefully result in something that makes lots of people happy.

– It is really interesting because this is that world that I want to live in where technology becomes the great equalizer so how do people that want to learn more about AI For Accessibility and the AI For Good initiatives, where do they go to learn more? And how do they get involved?

– We have a great website microsoft.com/aiforacessbility that tells more details around the grants that are available and the grantees that have been awarded already. And some information around how Azure can be involved with accessible technology development. And our grant application is at ak.ms/grant made it super simple.

– [Man] Simple’s good.

– And so anyone can apply anywhere in the world. If you have a product today that you want to explore whether or not AI can help we can support that. If you are an NGO that wants to work with some developers we can help with that too. So we definitely want to hear from the community to find out what this robust developer community can bring here.

– It’s been fun to hang out with you Mary in Microsoft’s Build conference. It’s incredible to see a company that I’ll say is doing the right thing to make technology again that great equalizer to make the lives of everyone on the planet better so thank you for the conversation it’s been super fun.

– [Woman] We’ve learned a lot from our guests about big data but now it’s time to get a bit personal. In a segment we like to call Rapid Fire. Pew pew.

– All right so what is the latest great book that you’ve read that you would recommend to our listeners?

– I have not read many books in quite some time I have been reading a lot of research papers. The book I am about to read is Chelsea Handler’s Life Will Be The Death Of Me. She’s coming to Seattle as part of her book tour. And so I am looking forward to reading that.

– Okay and then research paper you would recommend to our listeners?

– Yeah, Sadeek Razi who is a researcher at RIT wrote a great paper with his adviser Matt Huenerfauth around sign language modeling and so the idea of a person is signing and there is idea of a signing space and the fact that while I’m telling a visual story I can place items in that space. And so for the sign language geeks in the world that’s a paper I’d highly recommend.

– Interesting.

– That’s awesome. So we are at a conference so if you were speaking at the conference or if you had a song to play when you go on stage what would it be?

– Probably Panic At The Disco High Hopes.

– [Man] High Hopes, good choice.

– Like it. Okay so what piece of technology is currently making your life worse?

– My cracked screen on my phone.

– Oh no.

– Join the club.

– It’s really sad.

– Have you gotten any of the splinters in the finger yet?

– Oh yeah, it’s been a real struggle.

– Equally one of my favorite commercials it was a couple of years ago where it was like the broken phone and the guy was like, had all the bandaids on his fingers. Bill Hader was pretty great.

– My crack’s getting bigger and bigger and it’s spreading across the screen right now eventually I’m gonna have to get it replaced. So what is your biggest personal money pit right now?

– Soul Cycle.

– [Man] Good choice.

– Good one.

– That’s not a bad choice.

– My favorite way of working out.

– Excellent, I’m a but that’s fine.

– He just got me into it last week and it’s awesome–

– It’s also terrible.

– The music is louder and there’s more techno in Soul Cycle so if you like that. Come party with us.

– That would be fun. All right so what show are you binging on right now?

– Schitt’s Creek.

– Really?

– Okay.

– Oh my goodness.

– Which one is that?

– It’s on Netflix, it’s with Dan Levy and Eugene Levy. Did you ever see the movie Best In Show?

– I did.

– Okay so Eugene Levy’s in that. So it’s him and his son who have written this series and it’s hilarious.

– I’m going to add that one to the list. All right and then lastly where’s the next interesting place that you are going?

– I’m going home to New York at some point in June to see my family.

– [Man] Well good for you.

– Well Mary it has been super fun to have you on the Big Data Beard podcast and I wish you the absolute best of luck in doing and delivering AI For Accessibility with Microsoft.

– Thank you so much.

– [Man] Thanks for listening to the Big Data Beard podcast.

– [Woman] The music for this episode is by Andrew Bell. Check him out on iTunes or Spotify.

– [Woman] And be sure to smash that thumbs up button so we can keep the episodes coming. Until next time keep being awesome.