Describe the environment – operate it using natural language
For blind or visually impaired users, Apple is significantly expanding visual recognition by using VoiceOver to provide more detailed descriptions of images system-wide via Apple Intelligence. In the future, in live detection, you can press the action button to ask questions about what is in the camera viewfinder. With regard to voice control, iOS 27 improves the ability to describe controls in natural language – for example based on color, position or displayed content (“open the purple folder”, “click on the forward button”). The reader for operating aids is also intended to better capture and reprocess complex documents.
Subtitles, controlling wheelchair via Vision Pro
Another innovation is automatically generated subtitles for videos without existing subtitles. According to Apple, the transcription is done via on-device speech recognition on iPhone, iPad, Mac, Apple TV and Vision Pro. Speaking of Vision Pro: Apple also announced a new function for the headset to control compatible electric wheelchair drive systems. The headset’s eye tracking is used as an input method; Apple is initially supporting the Tolt and LUCI systems in the USA.
Many other innovations
In addition, smaller operating aids are coming to several platforms. visionOS receives vehicle movement alerts to combat motion sickness, facial gestures and new eye control options. tvOS will support larger fonts in the future, MFi hearing aids will pair more reliably and switch between Apple devices – and name recognition for deaf or hard of hearing users will work in more than 50 languages worldwide. For gaming, Apple also supports the Sony Access Controller on iPhone, iPad and Mac.