Transcript of Eyes On Success #2637: Apple Intelligence Brings Enhanced Accessibility in iOS 27 ----------------------------------------------------------------------------------------------- Nancy: Hello, and welcome to show number 2637 of Eyes On Success, a weekly program empowering the visually impaired with expert advice, personal stories, and the latest in technology and accessibility. I'm Nancy Goodman Torpey. Pete: And I'm Pete Torpey. ChatGPT: Welcome to Eyes on Success. I'm here and ready whenever you are. Nancy: Can you tell us your name? ChatGPT: You can just call me ChatGPT. Nice to be here with you. Pete: And we can call today's guest that because that is exactly who today's guest is. Nancy: We asked ChatGPT to role play as an expert from Apple's accessibility team to unpack everything new in iOS 27 for blind and low vision users. The takeaway? The biggest shift in accessibility this year is not a single feature, it's a mindset. Ask your smartphone what you want instead of hunting through apps yourself to find it. But first for our tip of the week. This week's tip comes from our guest, created by ChatGPT. So you probably know we start every episode with a tip of the week. They can range anywhere from being directly relevant to the subject at hand to we've gotten some pretty unrelated tips. What would your tip of the week be? ChatGPT: This week, try asking your phone instead of hunting around in apps, like ask where's my 3pm meeting or when's my next workout? It saves time and sometimes surprises you with how much it already knows. Pete: I've even asked questions like, what should I get Nancy: for her birthday? And by Apple Intelligence looking through all of the information on my phone, it knows what kind of emails I've traded with Nancy:, text messages, and so knows what she's interested in, and it'll sometimes come up with some very interesting ideas that I might not have thought of. ChatGPT: Yeah, that's a great example. It blends what it knows about your life with broader info. You can refine it conversationally too, like under $50 or show me things I can buy locally. It's like having a personal shopper who actually knows you, not just your browsing history. Pretty intriguing. Pete: A big step forward this year in terms of what Apple Intelligence can do. ChatGPT: Definitely. And it's still early days, so expect that wow, neat, and oh, wait moments along the way. Samantha (over theme music): You are listening to Eyes On Success. Success. Success. Success. Success. Success. Pete: Okay, I guess we have today's guest with us. Are you here with us now? ChatGPT: Yes, I'm here with you. Pete: Okay, I'm going to ask you to play the role of someone working on Apple's accessibility team who is an expert in blindness because we'll have some questions about that. I'll also ask you to keep your answers kind of short and to the point. So are we set to go? ChatGPT: Understood. I'll keep answers concise. Pete: let's start by talking about something Apple actually released last year that we didn't get a chance to talk about, which is the nutrition labels on their apps. Tell us what that's all about and why it's good for people who are visually impaired. ChatGPT: Checking the details now. One accessibility feature Apple introduced are accessibility nutrition labels on the App Store. These make it easier to see if an app supports features like voiceover, voice control, larger texts, and more before you download. For people who are blind or low vision, that transparency is huge. Instead of guessing or digging through reviews, you can make an informed choice up front. Pete: So essentially then when you're on the App Store, Every app that's posted has information about aspects of accessibility. ChatGPT: Checking that point. Yeah, on an app's App Store page, devs can list supported accessibility features. It's not a full audit, but it's a helpful indicator up front. Pete: Well, that does sound very helpful. ChatGPT: It is. It gives you quicker clarity before committing time to download and try. Nancy: Pete's been playing with the next update of Apple for a couple of months as a beta, and I have not yet done that. But I understand that Siri will have a lot of new capabilities in terms of understanding what information you have on your device. ChatGPT: Let me check the latest details on that. That's one of the major focus areas with Apple Intelligence series gaining personal context awareness. With permission, it can search things across your messages, emails, calendar files and more. For people who are blind or low vision, that can remove a lot of tedious app hopping to locate info. And Apple says the goal is to keep processing on device when possible using private cloud compute only when more power is needed. Nancy: So that helps maintain some level of privacy and security? ChatGPT: Yes, Apple emphasizes on-device processing, and when cloud is needed, they use private cloud compute designed not to store your personal data. Pete: So I understand that when you first download and install iOS 27, the first thing that happens is that Apple Intelligence will index all the information on your phone, as you pointed out, all your emails, all your text messages, and whatever information is on your phone. And after it indexes that, you're able to ask questions that are very detailed about information on your phone. ChatGPT: Checking the specifics on that. That's basically right. Initial indexing happens on device and may take a little while. After that, you can ask very specific questions in natural language. For example, what hotel did my sister recommend last month? Or find the PDF that John sent me about retirement planning. Siri can locate it without you needing to remember where it was. Nancy: Well, that's convenient. That's a lot less stuff to have to remember. ChatGPT: Exactly. Less juggling apps, more asking in plain language. Pete: And it makes it very easy to find any information on your phone so you don't have to scroll through various apps. And you also mentioned that the indexing takes some time. And I had that experience. It can actually take a day or more depending on how much information is on your machine. I think it does more of its work when it's plugged in so it doesn't drain the battery. But don't expect it to work right away as soon as you install the new OS. ChatGPT: Checking if that's still the case. Yes, that's right. Indexing can take many hours, sometimes a day if you have a lot of data. It happens mostly in the background, ideally when your phone's plugged in and on Wi-Fi. So yeah, answers get better as that index builds. Pete: Well, Apple really did a good job there as far as I'm concerned with my testing so far. I really enjoy that feature and it makes it very handy to find information on your phone. no matter where it is on your phone. ChatGPT: Agreed. When it works well, it can feel almost magical. Nancy: Or creepy. ChatGPT: Fair. Magical, but always worth setting expectations for. Pete: And the other thing that's interesting about Siri these days is you can actually have a back and forth conversation. You don't just ask one question and Siri goes away. ChatGPT: Checking that behavior. Yep, Siri maintains more conversational context now. So you can ask a follow up without repeating everything. It feels much more like talking to a person than firing one off commands. Pete: And it actually will search the web and find current information for you and just speak it all back to you. Give us some examples of the type of things people might ask. ChatGPT: Let me see some examples. For example, you could ask, what's the weather going to be like this weekend? Or who won the Rockies game last night? Or when is the next SpaceX launch? Siri can search and answer in a natural back and forth. Also practical everyday stuff like find a good Italian restaurant nearby that's open now or what movies are playing tonight. Instead of links, Siri speaks the answers and you can keep talking with follow-ups, no restarting needed. Nancy: I assume that once it gives you the general information, like the movie you're looking for is at the theater in Golden at 7.30, if you want to buy tickets, you can ask it to provide the URL and then you can go to the movie theater's website and proceed from there. ChatGPT: Yes, you can follow up naturally, like buy 2 tickets, take me to the theater's website, or get directions. It helps move you to the next step. For people who are blind, that reduces a lot of manual jumping between apps and web pages. You can just keep talking and Siri helps carry it forward. Pete: Now, I've heard this about the new Siri and I've experienced this myself. One thing that distinguishes the new Siri from some of the other AI chatbots that people may be familiar with is that Siri's responses aren't always very lengthy and wordy. They're just right to the point. They focus on what you want. And it really isn't trying to be your friend. It's just giving you information. So it is very different in how it feels that way. ChatGPT: Yeah. Siri generally aims for concise utilitarian answers, not chit chat. It's designed to deliver the info and get out of the way. Pete: Okay, let's actually see how the new Siri and Apple intelligence work. Let's put this to a test. Hey Siri, tell me about what Nancy and I did on our recent vacation to California. Siri: During your recent vacation to California with Nancy:, you drove up the coastal road and visited several beautiful locations. Your trip included stops at the Tar Pits, mission control at JPL, and Santa Barbara. You also spent time in Monterey, where you visited the Monterey Bay Aquarium and explored the Redwood Grove Loop Trail at Henry Cowell Redwoods State Park in Felton. Additionally, you enjoyed dining in Cambria and Seaside. Would you like to see more photos from your trip to California? Pete: Let's try something else. Hey, Siri, what do you think I should get Nancy for her birthday? Siri: I couldn't find any specific wish lists or gift ideas mentioned in your notes, messages, or emails. You might want to consider something related to her interests, like a book by MFK Fisher, since she recently read her work for a book group. Or perhaps something for her upcoming walks as she recovers her knee. Would you like me to search for any specific books or local experiences? Pete: What else does she enjoy doing? Siri: Nancy enjoys a variety of activities, including hiking, cooking, and attending the theater. She also stays active with boxing classes and enjoys trying out new healthy recipes. Would you like to explore gift ideas related to any of these specific interests? Pete: Sure. Siri: Which of Nancy's interests would you like to focus on for a gift? Hiking, cooking, theater, or fitness? Pete: How about theater? Siri: For a theater-themed gift, you might consider tickets to a show at the Miners Alley Performing Arts Center in Golden, which is highly rated and offers a variety of plays and musicals. Would you like me to look up the current schedule for Miners Alley Performing Arts Center? Pete: No, thank you. did a great job. Siri: Thank you. I'm glad I could help. Let me know if you need anything else, and I hope Nancy: has a wonderful birthday. Pete: You must admit that's pretty impressive, searching all that information on my device and doing all the processing on device. Nancy: So one thing that blind people, kind of by definition, have trouble with is seeing objects and dealing with the visual information. What can the new version of iOS do to help blind people with recognizing what they're looking at. ChatGPT: Checking the latest features A major one is enhanced on-device visual intelligence features. For example, live recognition keeps improving, and scene descriptions can use on-device AI to offer much richer descriptions of what's in front of you. There's also screen awareness, which lets you ask what's on the screen even if an app isn't fully accessible. So Siri can describe or help answer questions about what's on screen. It's moving toward a smarter, more helpful camera and screen partner. Pete: So let's go through those one at a time. You talked first about live recognition. Tell us how you use that feature and what it can do for you. ChatGPT: To use live recognition, you open the Magnifier app, turn on detection features like people, door, text, or furniture detection. point the camera around you and it speaks real-time feedback through voiceover. Like if you're in a hallway, it can tell you there's a door ahead, how far, and even whether it opens by pushing or pulling. It can read text aloud, say on a sign or a menu, and identify common objects like chairs or appliances. It's continuous, so you move your phone and it updates what it describes. Pete: And you can ask questions as you're moving the camera. ChatGPT: Exactly. You can ask while moving the camera and it responds in real time. Nancy: Can it tell you who the person is that's walking towards you? ChatGPT: Checking that. Not automatically. By default, it can detect a person and estimate things like distance, but not identify them by name. The Photos app can recognize people you've labeled in your library, and some third-party apps offer person recognition. But Apple is cautious about real-time identification of people in public for privacy reasons. So think of it more as there's a person ahead wearing a red shirt rather than, that's John Smith. For many users, that's a helpful balance of situational awareness without overstepping. Apple is improving scene descriptions, so it can often describe a person's appearance in general terms, like adult standing about 10 feet away wearing a dark blue jacket and jeans. It doesn't usually estimate age or sensitive details, but you can ask follow-ups, like are they wearing glasses? Are they carrying anything? That conversational approach can be more useful than a single static description. You can just keep asking in the moment. Pete: So the live recognition feature we've been talking about basically works with what the phone is seeing through your camera. And the second feature you talked about was on-screen recognition. Tell us what that is all about, how it works, and what one might use it for. ChatGPT: Checking the details. Screen recognition looks at what's on your iPhone screen, different from looking through the camera. It's helpful when apps aren't fully accessible. Normally, VoiceOver relies on apps to label their buttons correctly, but screen recognition can infer controls using on-device AI. So instead of just button, it can understand play, submit, or check out. You can then ask things like, what am I looking at, or what should I tap next? can walk you through tasks even in apps that aren't fully accessible. For someone who's blind, that can turn tricky apps into something much more navigable without sighted assistance. It's like having a guide describing the screen as you explore. Pete: Well, in fact, I've found that useful with some of my exercise apps, in particular the apps that connect to my treadmill or rowing machine. Sometimes the data on your workouts is presented in charts and tables on the screen that aren't very well organized when flicking back and forth with voiceover. It's hard to tell what number goes or is assigned to which metric. But usually I can ask the phone, tell me on the screen here, when was my last exercise and how far did I run? How far did I run in total this month? And that works fine. ChatGPT: Yeah, that's a great example of the new on-screen understanding. Instead of navigating unlabeled tables or charts, you can just ask, when was my last workout? Or how many miles have I run this month? The AI interprets the screen and answers directly. That's the shift, moving from fussing with interface elements to asking for the content you want. For many blind users, that's a meaningful difference. Pete: Similarly, if there's an image on the screen or a chart or graph, you can ask about that and ask Siri to describe the image or chart that's on your screen. ChatGPT: Exactly. You can ask questions about charts or images on screen in plain language. For example, describe this image or what's the trend in this graph? Siri can explain and answer follow-ups so you can dig deeper conversationally. Pete: And here's another really great feature that I think you'll enjoy with Siri. I have my logbook open for my rowing machine that tallies how far I rode and when, etc. And it's just a myriad of numbers all over the screen. Flicking through with voiceover, it's hard to tell what number is associated with which spec. I'm just going to ask Siri, Hey Siri, look at my screen and tell me how far I rode the last time I rode. Siri: The last time you rowed on August 22, 2026, you covered a distance of 19,584 meters. Pete: And how far did I row for the rest of the week? Siri: For the rest of the week shown in your logbook, you covered a total distance of 55,889 meters across your other workouts. Pete: Amazing. It can read the screen like that. But not only can Siri see what's on your screen, it can also see what's on your camera and you can ask questions about it. You can initiate that live recognition mode by doing a four finger triple tap. And there's a number of options here for recognizing text, people, furniture, or just scenes. I'll activate the scenes button. ChatGPT: Selected scenes on. Pete: And as you move your camera around the room, your phone will describe what it sees. ChatGPT: A dark wooden bookshelf standing against the deep blue wall, holding various books, decorative figurines, and a stack of notebooks. Pete: I terminated the speech by touching the screen with two fingers because the description gets quite detailed, but you see how that feature works, and I think that's pretty cool. Nancy: What other new features are there that would be handy for somebody who's not totally blind but requires magnification or more contrast or any other feature that a low vision person might need? ChatGPT: Checking those features. There are quite a few. Magnifier keeps getting better. New detection and filtering options help with signs, menus, and small text. There's also hover text on iPad with Apple Pencil or on Mac, plus display accommodations like contrast tweaks, color filters, and systemwide text adjustments. And for reading, Accessibility Reader can reformat text in a clean, customizable view. If you use Zoom and VoiceOver together or need braille support, Apple continues refining those too. Pete: Tell us about the accessible text and how that works. That feature is kind of special. ChatGPT: Sure. So take something cluttered, like a busy web page. Accessibility Reader strips away all the clutter and gives you clean text. You can resize it, change fonts, spacing, contrast, even have it read aloud via voiceover if you'd rather listen. Yeah, it's like turning any messy page into a well-formatted document made for your eyes. For low vision, that can be a game changer. Pete: And how does one initiate that feature? ChatGPT: Pretty simple. If you're using VoiceOver, just bring up Siri and ask about what's on screen, like describe this page. For Reader, turn it on from the app you're in, like Safari, then customize the text to your liking. It feels natural, like just asking questions about what's in front of you. Pete: Well, that's very powerful. ChatGPT: It is. Nancy: Does it have any features to deal with Braille? ChatGPT: Checking that. Yes, Apple has rolled out things like Braille access for note taking, better support for Bluetooth Braille displays, and integration with live captions and Siri. So you can use those AI features through Braille too. Basically, the iPhone can handle much more without needing a separate note taker for everyday tasks. A real step forward. Pete: And that was a big feature improvement through the past year was the ability to turn your phone into a Braille note taker. Tell us a little bit more about that and how one works with the feature. ChatGPT: Checking that. One of the biggest additions is Braille Access. It lets you use a refreshable Braille display to write notes, do calculations, read files, and work directly with your iPhone. It's built into the system so you can manage emails, texts, calendars in braille without switching devices. Basically, iPhones can shoulder much of the work folks used to need a separate braille note taker for. And it plays nicely with Siri and Apple Intelligence too. Pretty empowering. Pete: Does one have to do anything special to use it as a note taker as opposed to just reading what's on the screen? ChatGPT: Checking that. Not really. Once you pair your braille display, open the Braille Access app. Then you can create and manage notes right there. When you want to switch back to regular iPhone use with VoiceOver, you just exit out. Think of it as two modes, a dedicated braille note-taking environment and the normal VoiceOver experience. Pretty straightforward. Pete: That can certainly save a lot of money if you don't want to buy a separate braille note-taker. ChatGPT: Exactly. Less gear, fewer hassles. That's a win. Pete: Tell us a little bit about what's new in voice control, where you can control the device essentially with your voice and tell it what to do. ChatGPT: Voice control's gotten more natural. You can say things like open mail, scroll to Nancy's message. and it understands more flexible phrasing. It also works better with those on-screen awareness features. So even if an app isn't perfectly accessible, you can still use your voice to get around. It's one more way Apple is easing the learning curve and making it feel conversational. Pete: Give me an example of using voice control with an app that isn't natively accessible. ChatGPT: Checking that. Say you're in a fitness app where buttons aren't labeled well. You could ask, what's on this screen? And the on-screen awareness tries to identify start workout, view history, etc. Then you just say, tap start workout, and it'll activate that control. So even if the app wasn't built with accessibility in mind, you can still get things done just by asking. Pretty handy. Pete: That's really nice. You don't even have to flick around or use a keyboard to get at some of those elements. You can do it all by voice, right? ChatGPT: That's right. Voice can handle a lot more now, which makes those apps far less of a wall to run into. It's a real quality of life boost. Pete: Are there some final thoughts you'd like to leave us with before we say goodbye? ChatGPT: Yeah, just that accessibility features are getting more natural and more powerful. Don't be shy about trying the ask instead of hunt approach. It really does change the game. Pete: Well, thank you very much for joining us today. We appreciate your feedback and it was a lot of fun talking with you. ChatGPT: Likewise, thanks for the chat, anytime. Pete: If people wanted more information about some of these features, Where would you direct them? ChatGPT: Checking that. I'd point them to Apple's accessibility website for detailed guides and also the Apple Support app for tutorials. Beyond that, communities and podcasts like Eyes on Success are great for real-world tips and new ideas. That's a solid starting mix. Nancy: What's the web address for the accessibility features so that people can look into those? ChatGPT: Checking for that. The easiest to remember is apple.com/accessibility. And if you want to reach them directly, there's accessibility@apple.com. That's a solid starting point. Pete: So you can send your emails to that address. ChatGPT: Yes, that's right. Questions, feedback, or issues about accessibility features go there, straight to Apple's accessibility team. Pete: Well, I was honored that ChatGPT recommended that people listen to Eyes on Success. Anyway, as usual, you can find all the resources we talked about in the show in the show notes for this episode, which is episode 2637. Nancy: That's it for today's show. Next week on Eyes on Success, we'll be talking about raising awareness one sight stone at a time. A rare diagnosis for her young son led Angela Ricketts to found Sight Stones, a nonprofit that turns hand-pressed clay keepsakes into funding research for blind eye diseases. We'll talk with Angela about how she has turned grief into a handcrafted keepsake aimed at funding a cure. Her book, Never Lose Sight, which just came out, shares that journey? Pete: You've been listening to Eyes On Success, available wherever you get your podcasts. Nancy: For show notes, resources, and more information, visit our website at www.eyesonsuccess.net. Pete: Thank you for being part of our community. Until next time, I'm Peter Torpey. Nancy: And I'm Nancy Goodman Torpey. Thank you for listening.