Beyond the Screen: How Smartphone AI Could Transform Accessibility for People With Vision Loss
August 24, 2026 | Health Technology & AI
Smartphones are usually discussed as sources of information, communication and entertainment. But a new development from researchers at Harvard University shows how the same device could potentially become something much more practical: an assistive navigation tool for people who are blind or have low vision.
Researchers at the Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS) have developed Mobilio, a smartphone application that combines artificial intelligence, built-in phone sensors and personalized audio guidance to help people navigate outdoor environments.
The research, published in Nature Biomedical Engineering, explores how an ordinary smartphone can provide continuous path guidance, turn-by-turn directions and obstacle avoidance without requiring a separate specialized navigation device.
The early results are notable. In tests involving 14 participants with blindness or visual impairment, people using Mobilio together with a white mobility cane completed an outdoor route about 13% faster than when using Google Maps and a cane. During an indoor obstacle course, participants also had about 41% fewer contacts with obstacles.
The technology is still a research project rather than a replacement for established mobility tools. Nevertheless, it demonstrates how AI and smartphones could work together to address a very real accessibility challenge.
Why Independent Navigation Matters
For someone who is blind or has significant vision loss, navigating an unfamiliar outdoor environment can involve challenges that conventional mapping applications are not designed to solve.
A digital map can tell a person where a street is located and provide directions from one destination to another. But a pedestrian also needs information about the immediate environment.
- Is the person still on the sidewalk?
- Where does the walkable path continue?
- Is there an obstacle directly ahead?
Has the pedestrian moved away from the intended path?
These are different problems from simply determining a person's GPS location.
Harvard researchers first surveyed more than 100 people who were blind or had low vision to understand what they wanted from navigation technology. The researchers identified three important capabilities: reliable turn-by-turn directions, continuous guidance along sidewalks and paths, and obstacle detection and avoidance.
Existing tools can address some of these needs, but not necessarily all of them at the same time.
A white mobility cane, for example, remains an important tool for detecting nearby obstacles and changes in the ground. Guide dogs can provide sophisticated assistance but require extensive training and ongoing care. Electronic travel aids can provide additional information but may require specialized hardware.
The Harvard team therefore explored a different question:
Could a smartphone that many people already own provide some of these capabilities through software?
What Is Mobilio?
Mobilio is designed as a smartphone-based navigation assistant for people who are blind or have low vision.
Rather than relying on a single sensor, the system combines information from several components of the phone.
These include:
- The smartphone camera
- GPS
- The phone's inertial measurement unit, which detects motion and orientation
- LiDAR when it is available on the device
- Computer-vision algorithms
- Personalized audio feedback
The researchers describe this combination as a form of sensor fusion. Information from different sensors is processed together to help the system understand both the user's location and the environment around them.
The camera is particularly important.
Instead of simply asking the phone where the user is on a map, Mobilio analyzes the environment from the perspective of a person walking through it.
That distinction is important because pedestrian environments contain details that conventional GPS navigation does not necessarily understand.
Teaching AI to See the World From a Pedestrian's Perspective
One of the interesting technical challenges behind Mobilio is that many existing computer-vision datasets are designed around images captured from cars.
A camera mounted on a vehicle sees roads and streets from a very different perspective from a person walking along a sidewalk.
For someone walking, the system needs to distinguish features such as:
- Sidewalks
- Crosswalks
- Roads
- Obstacles
- Other parts of the pedestrian environment
Raymond Liu, the Harvard Ph.D. student who led the research, trained a semantic-segmentation model using pedestrian-view images so the system could better recognize the environment from a walking person's perspective.
This illustrates an important point about AI.
Having a powerful model is not always enough. The model also needs appropriate data representing the environment in which it will operate.
How the Smartphone Becomes a Navigation Assistant
Mobilio essentially combines several sources of information.
The Camera
The camera provides continuous visual information about the environment.
Computer vision can analyze the scene and identify features that are relevant to walking.
GPS
GPS provides information about the user's larger geographic position and helps with route planning.
However, GPS alone is not sufficiently precise for every pedestrian-navigation task.
Motion Sensors
The smartphone's inertial measurement unit provides information about movement and orientation.
This can help the system understand how the user is moving and how their position is changing.
LiDAR
On smartphones equipped with LiDAR, the system can also use depth information.
This is not available on every smartphone, so the technology has been designed around the sensors that are actually present on a particular device.
Together, these different inputs allow Mobilio to build a more detailed picture of the user's situation.
Why Mobilio Uses Sounds Instead of Constant Speech
One of the most interesting parts of the project is the way information is communicated to the user.
A conventional navigation application might continuously speak instructions such as telling a user to turn left or right.
But for someone who is blind, hearing is already an important source of environmental information.
Sounds from vehicles, pedestrians and other surroundings can provide valuable clues about what is happening nearby.
Too much spoken information could therefore become distracting.
Mobilio instead uses personalized audio beeps.
The sounds can indicate the direction in which the user should move. A cue coming from one side can encourage the person to adjust their direction toward that side.
Harvard describes this as personalized audio feedback rather than simply giving users a stream of spoken instructions.
The System Learns How Each Person Responds
Another important part of the technology is personalization.
People do not all respond to sound in exactly the same way.
One person may respond quickly to a particular audio cue, while another may require a different pitch or timing.
Mobilio therefore uses what the researchers describe as human-in-the-loop optimization.
The system observes how accurately a person follows the audio guidance and adjusts the cues to better suit that individual.
This means the system is not simply delivering exactly the same audio pattern to everyone.
Instead, the technology attempts to adapt its feedback based on the user's responses.
What Happened During the Tests?
The Harvard team evaluated Mobilio with 14 volunteer participants recruited through the Carroll Center for the Blind in Newton, Massachusetts.
Participants completed navigation tasks using two different setups.
The baseline condition involved:
Google Maps + a white mobility cane
The experimental condition involved:
Mobilio + a white mobility cane
Participants navigated both an outdoor community path and an indoor obstacle course designed to represent common environmental challenges.
The results were encouraging.
Outdoor Navigation
Participants using Mobilio and a cane completed the outdoor route approximately 13% faster than when using Google Maps and a cane.
Obstacle Navigation
During the indoor obstacle course, participants using Mobilio and a cane had approximately 41% fewer contacts with obstacles.
Harvard also reported that Mobilio performed comparably to a human guide in terms of reliability during the study.
These findings do not mean the technology has been proven to replace a human guide or established mobility tools in every situation. The study involved a relatively small group of participants and controlled testing environments.
Instead, the results provide evidence that smartphone-based AI navigation deserves further research.
A Smartphone Could Be More Accessible Than Specialized Hardware.
One of the strongest ideas behind the project is accessibility.
Specialized assistive devices can be expensive and difficult to manufacture at scale.
A smartphone, on the other hand, is already a widely available consumer device.
The researchers specifically wanted to determine whether existing smartphone hardware could provide useful navigation capabilities without requiring users to purchase an entirely new machine.
Raymond Liu explained that previous specialized technology could be effective but expensive and difficult to scale. The smartphone-based approach offers the possibility of delivering assistance through a device many people already possess.
This could be particularly important in regions where access to specialized assistive technology is limited.
However, affordability cannot be assumed simply because an application runs on a smartphone. Device compatibility, internet access, smartphone costs and accessibility of the software itself can all affect who can benefit.
Could This Become a Public-Health Tool?
The most interesting aspect of Mobilio may be that its potential benefits extend beyond navigation technology.
Mobility is closely connected to independence and participation in everyday life.
Being able to travel to school, work, shops, healthcare appointments or social activities can make a major difference in someone's ability to participate in society.
A navigation system that helps a person move more confidently through unfamiliar environments could therefore have broader implications for accessibility.
But it is important not to overstate the medical implications.
Mobilio is not a medical treatment, and the Harvard study does not establish that the app prevents injuries, improves cardiovascular health or treats psychological conditions.
Its demonstrated purpose is navigation assistance.
Any broader health benefits would need to be studied separately.
Could This Technology Work in Africa?
The idea is particularly interesting when considering regions where specialized assistive technologies may be difficult to access.
Smartphones have become important tools for communication, education, banking and access to information in many African countries.
If sophisticated accessibility functions can be delivered through existing smartphones, software could potentially expand access to assistive technology without requiring every user to purchase a dedicated device.
There are still major challenges.
Reliable GPS is needed for some navigation functions. Smartphones must have sufficient processing capabilities, and users need access to appropriate devices and software.
Urban environments can also differ substantially from the locations where an AI navigation system was trained and tested.
A system developed using pedestrian environments in Massachusetts, for example, would need additional evaluation before claims could be made about performance in cities elsewhere.
This is why Harvard's next phase of research is important.
What Happens Next?
The research team plans to test Mobilio in a wider range of realistic environments around the Boston area.
The researchers also want to understand how well the technology performs in different environments and how naturally it can fit into people's everyday routines. Harvard's Grid Accelerator is supporting efforts to help translate the technology toward broader accessibility and potential commercialization.
The Harvard Office of Technology Development has also protected innovations associated with the research and is pursuing commercialization opportunities.
That means Mobilio is still at an important transition point between academic research and potential real-world deployment.
What Are the Limitations?
Despite the promising results, Mobilio should not be viewed as a perfect navigation system.
The study involved only 14 participants, which is a small sample for drawing conclusions about how the technology will perform across the wider population of people with blindness or low vision.
People have different levels and types of visual impairment, different walking habits and different environmental needs.
The researchers themselves emphasize the diversity of blindness and visual impairment and the importance of testing assistive technology with actual users.
There is also the issue of unusual environments.
An AI system trained on particular types of pedestrian infrastructure may encounter difficulties when faced with environments that differ significantly from its training data.
Weather, lighting, crowded streets, construction, unusual road layouts and other unexpected conditions can also create challenges for computer-vision systems.
These are reasons why further testing is necessary before such technology can be considered a mature replacement for existing navigation assistance.
Why Human Testing Matters
One of the strongest aspects of the Harvard research is its emphasis on testing with people who would actually use the technology.
Assistive technology cannot be evaluated only by looking at technical performance.
A system may work well according to a computer benchmark but still be difficult or uncomfortable for a real person to use.
The researchers therefore incorporated feedback from participants as part of the development process.
Liu has also spoken about his personal motivation for the project: his older brother is blind, which helped shape his interest in developing technology that could support independent navigation.
That personal connection illustrates an important principle in health and accessibility technology: the people who use a device should be involved in determining whether it is actually useful.
The Bigger Picture: AI Is Moving Into the Physical World
Mobilio represents a broader direction in artificial intelligence.
For years, many AI applications were primarily digital. Users typed something into a computer or phone and received information in return.
Modern AI is increasingly interacting with the physical environment.
Cameras allow systems to see.
Microphones allow them to hear.
Motion sensors allow them to understand movement.
GPS provides location information.
Together, these technologies allow software to respond to the world around the user.
Mobilio demonstrates how these capabilities can be combined for a specific human need rather than simply being used as a technological demonstration.
Tech4Health Tips Takeaway
The most important lesson from Mobilio is not that smartphones will replace canes, guide dogs or human assistance.
Instead, the research demonstrates what can happen when AI, ordinary consumer hardware and accessibility-focused design are brought together.
A smartphone already contains several sensors capable of measuring location, movement and the surrounding environment. Researchers at Harvard have shown that these components can be combined with machine learning and personalized audio feedback to support navigation for people with blindness or low vision.
The early results are promising, but further testing is needed.
For Tech4Health Tips readers, Mobilio is a good example of how health technology does not always have to involve a new medical device.
Sometimes, innovation comes from using a device that people already carry in their pockets and finding a new way to make it useful.
As artificial intelligence continues to move beyond chatbots and into cameras, sensors and everyday devices, accessibility could become one of the areas where these technologies have their most practical impact.
Frequently Asked Questions
What is Mobilio?
Mobilio is an AI-driven smartphone application developed by Harvard researchers to assist people who are blind or have low vision with outdoor navigation.
Who developed Mobilio?
The research was led by Harvard Ph.D. student Raymond Liu in collaboration with Patrick Slade, an assistant professor of bioengineering at Harvard SEAS.
How does Mobilio work?
The application combines smartphone camera data, GPS, motion and orientation sensors, and LiDAR when available. It processes this information to provide navigation guidance through personalized audio cues.
How much faster was Mobilio in the study?
Participants completed the outdoor route approximately 13% faster using Mobilio and a cane compared with Google Maps and a cane.
Did Mobilio reduce obstacle contacts?
Yes. In the indoor obstacle course, participants using Mobilio and a cane had approximately 41% fewer contacts with obstacles than in the comparison condition.
Can Mobilio replace a guide dog or mobility cane?
The research does not establish that. Mobilio was tested alongside a white mobility cane, and further research is needed to determine how the technology should be used in everyday life.
Is Mobilio available to everyone?
The Harvard research describes Mobilio as a research project being developed toward potential broader accessibility. It should not currently be presented as a universally available consumer navigation application.
Conclusion
Harvard's Mobilio project offers a compelling example of how artificial intelligence can be used for something more meaningful than generating text or images.
By combining a smartphone's camera and sensors with computer vision and personalized audio feedback, researchers have developed a system that can help people with blindness or low vision navigate outdoor environments.
The early study produced measurable improvements in navigation time and obstacle contacts, but the technology still requires broader testing.
Its greatest significance may ultimately lie in the idea behind it: powerful assistive technology does not always require an entirely new device. Sometimes the solution can begin with the technology already in someone's pocket.
Sources
- Harvard John A. Paulson School of Engineering and Applied Sciences — A Smartphone Navigation App for People with Blindness and Low Vision.
- Nature Biomedical Engineering — Improving outdoor navigation for people with blindness using an AI-driven smartphone application and personalized audio guidance.
- Harvard Community and Campus Life — background on Raymond Liu and the development of Mobilio.