When will Android have general screen reader functionality?

22 views
Skip to first unread message

blindfold

unread,
Sep 27, 2009, 2:24:49 AM9/27/09
to TTS-for-Android
Hi,

I find that increasing numbers of blind people are moving over from
Nokia Symbian phones running the Talks or Mobile Speak screen reader
to the Apple iPhone with its built-in VoiceOver speech for general
phone accessibility. AFAIK there are still virtually no blind users of
Android phones for lack of this level of accessibility. After a number
of unanswered inquiries, there is still no indication that there will
come a screen reader for Android? IMO leaving accessibility to
developers of point applications by making a built-in TTS available
will not fly. A speech shell for launching applications is also not
enough: one needs general speech access to all common GUI elements
across and within all applications, not just those applications that
were developed with accessibility in mind. Any ideas or announcements?

Thanks,

Peter Meijer


The vOICe for Android
http://www.seeingwithsound.com/android.htm

Per

unread,
Sep 29, 2009, 8:00:05 AM9/29/09
to tts-for...@googlegroups.com
I would like to understand: What can be done with the new accessibility
framework APIs and what not? What else is needed for a general Android
screen reader?

Unfortunately, I already have understood that Android OS developers do not
publicly report about their work, plans and strategy. The schedule for
possible future accessibility probably depends on work progress and the
strategy for the whole Android platform, but I still trust in Google's
rationality.

There are legal reasons such as Section 508 of the US Rehabilitation Act and
similar future European standards. Coming mobile devices must be accessible
respectively 508 compliant and this depends on the OS. I am sure, Google
will do its homework, but I'd also like to know when this will happen.

Fortunately, Apple already has added high-quality built-in eyes-free access
to its products, but the current work of Apple's huge accessibility team is
non-public. RIM is working on eyes-free access, but blind users will have to
buy a very expensive third-party screen reader software first before they
can use the new Blackberrys. At the moment, there is no need to speak about
Windows Mobile and Nokia/Symbian, they are behind. For example, follow the
link of Nuance's screen reader dev to his post "Accessibility in QT and
beyond" in the comment section of this post on Symbian's official blog:
http://blog.symbian.org/2009/09/15/can-mobile-be-accessible/

In my opinion, Google should choose an alternative way. Making the whole
Android accessibility development process as public as possible, like every
open source project should be anyway. Google and/or the Open Handset could
invite interested parties and professionals to form an open Android
Accessibility team, involve qualified users and make Android accessibility
efforts always visible for the public.

I wish there would be an official paper which describes the accessibility
strategy for Android. At least, Google could publish a kind of statement on
this issue. Unfortunately, as it seems, Google employees are not allowed or
willing to provide detailed information about Android's Accessibility. That
is a pitty.

Best regards, Per
http://twitter.com/PerBusch

Dominic Mazzoni

unread,
Sep 29, 2009, 10:23:47 AM9/29/09
to tts-for...@googlegroups.com
Peter,

Thanks very much for your questions about Android accessibility. I
work at Google and I'm a contributor to Android's accessibility
efforts. Here's what I can tell you so far:

* The Android 1.6 SDK takes first steps by introducing system-level
support for TTS and adding a framework for developers to build
accessibility software like screenreaders.

* There are people at Google working on this.

* There is still a lot of work to do and it will be some time before
Android will be able to provide an excellent experience for users with
disabilities.

Please understand that I can't say much more at this time, but we are
listening and we very much appreciate your interest and support.

- Dominic

Nolan Darilek

unread,
Sep 29, 2009, 5:23:46 PM9/29/09
to tts-for...@googlegroups.com
I've been playing with the new Donut accessibility API for the past few
days. A few points to keep in mind:

1. I'm totally blind, so my interactions with the emulator are a bit
rough. Specifically, I have AccessibilityEvents being output to an adb
logcat, which spits out stuff on my terminal. As you can probably
imagine, this is rather suboptimal. :)

2. I'm new to Android development. While it would make sense to start
with "Hello, world," it's rather useless if you can't read the screen to
begin with.

Thus far, I have many of the UI elements providing some level of spoken
feedback. In the emulator, I can navigate the stock launcher screen,
change settings, perform searches, etc. Spoken feedback is provided, but
is rather jerky, and in order to achieve some level of fluidity, I had
to make sacrifices (I.e. since there is apparently no way to globally
intercept keypresses, I have to manually stop TTS on certain
AccessibilityEvents, though this has the unfortunate issue of not
speaking some dialogs because the focus event kills text that would have
been spoken by other events.) So far I've had some luck with the music
player and email app as well.

There's more work to be done. Specifically, none of WebKit appears
accessible. I get some events, but none include meaningful text, and I'm
not sure whether I'm only moving between focusable elements or between
lines of text on the page. I'm also not sure that there's any way to
access a hierarchy of widgets for more advanced presentation. So, for
instance, while I can successfully capture the fact that I've focused a
text area with "143" in the email application, communicate that this is
an EditText widget, provide meaningful feedback on update and while I
know that this is my IMAP port number, I can't figure out how to
determine which label might be associated with this field, or even how
to provide screen review functionality. Seems like I should be able to
capture an AccessibilityEvent with fullScreen==true, but this doesn't
seem nearly dynamic enough to capture the location of potentially
dynamic widgets in that text, and doesn't seem to include the full
screen contents as I'd expect.

I'm also having trouble figuring out whether or not this is even
possible, since I don't know of any accessibility-specific communities
to ask. My questions here and on the main android-dev group have gone
unanswered, and what I've achieved so far has been done on my own.

But there's at least a start, and I can at least navigate the emulator
now. I've even got the beginnings of an architecture for making existing
apps more accessible in instances where buttons and such aren't labeled.
It isn't really scripting, since the code needs to be compiled in right
now, but it's at least a nice DSL for the task of mapping specific
speech to specific AccessibilityEvents from a given package/class.

Also wishing Google would be more open about this. I'd love to play with
this technology, but unless I'm missing something, my next phone will
likely be something else. :( Hopefully in another year or so...
--
Nolan Darilek
http://thewordnerd.info

blindfold

unread,
Sep 30, 2009, 4:56:49 AM9/30/09
to TTS-for-Android
Thank you for your reply, Dominic. It is at least good to know that
you cannot say more at the moment, but that you do hear our concerns.
In the longer term I should be able to remove most of the TTS-based
accessibility features of my app once general and affordable/free
screen reader functionality becomes available, because the explicitly
programmed full accessibility of menus and dialogs within my app is
only meant as a temporary stopgap. Only the event-based spoken
messages will then remain, because a screen reader cannot know about
them unless I would give it rather obtrusive "hints" such as through
popup messages. However, the latter cause awkward timing issues and
compete with how one may want to present events visually. (I tried it
once for my Java ME MIDlet with Nuance Talks running on Nokia phones,
but it did not work reliably because I had to guess when the screen
reader would have detected and spoken my message before letting my
popup time out, while popups tend to be annoying to the sighted.)

Thank you for your efforts!

Peter
On Sep 29, 4:23 pm, Dominic Mazzoni <dmazz...@google.com> wrote:
> Peter,
>
> Thanks very much for your questions about Android accessibility.  I
> work at Google and I'm a contributor to Android's accessibility
> efforts.  Here's what I can tell you so far:
>
> * The Android 1.6 SDK takes first steps by introducing system-level
> support for TTS and adding a framework for developers to build
> accessibility software like screenreaders.
>
> * There are people at Google working on this.
>
> * There is still a lot of work to do and it will be some time before
> Android will be able to provide an excellent experience for users with
> disabilities.
>
> Please understand that I can't say much more at this time, but we are
> listening and we very much appreciate your interest and support.
>
> - Dominic
>

Bernard Segonnes

unread,
Oct 1, 2009, 11:51:20 AM10/1/09
to tts-for...@googlegroups.com
Hi,

I understand you can not say every thing :-)

But at least : is it planned (no date, just yes or no) to add TTS for other languages : chinese, japanese, russian, korean, portugese, etc

Those TTS languages should be a bit hard to program, but there are customers for our Android App. in those countries.

Dominic Mazzoni

unread,
Oct 2, 2009, 11:53:50 AM10/2/09
to tts-for...@googlegroups.com
Bernard,

There will definitely be TTS support in other languages.  Exactly which languages and when depends a lot on the availability of free or licensable speech engines and high-quality voices for those languages, but Android is a global product, and we understand that accessibility means supporting users around the world no matter what language they speak.

- Dominic

Nolan Darilek

unread,
Oct 2, 2009, 10:53:38 PM10/2/09
to tts-for...@googlegroups.com
I just forced my attempt at an Android screen reader onto a few
unsuspecting victi...er...friends, and thought I'd toss it out here.
Keep in mind that this is the result of about a week's worth of work.
It's not great, and I'm not sure how much better I can make it right
now, but I can navigate most of the screens I've tried, more or less,
with the exception of the browser. There are a bunch of unlabeled
buttons, particularly in the dialer, but once I sit down with someone
who can see I plan to script it, plus the various other apps I find with
unlabeled controls. The .apk is here:

http://dl.getdropbox.com/u/147071/spiel-1.0.apk

The source sans build instructions is here:

http://gitorious.org/spiel

You'll have to enable installing unsigned applications via
Settings->Applications. Then, once installed, click Settings,
Accessibility, Spiel. Click OK at the scary warning and, assuming you
have Donut and TTS, you should start getting speech.

If you're brave enough to test this, here are a few questions I'd like
answered:

1. How well does it perform? I can't compare emulator performance with
vs. without it running, and I'm not sure how emulator performance
compares with phones themselves, so I'm curious. Not sure what to do if
performance sucks, I've got some embarrassing ugly hackery in place to
deal with the fact that I can't stop speech automatically on keypress,
unless said keypress triggers an ON_TEXT_CHANGED.

2. If you move your finger along the touchscreen without tapping, are
items spoken as they're hit? They are in the emulator when I use arrows,
but arrows don't grant access to the notification area. A lot of my
perceptions of API limitations spring from its inability to let me
access a widget hierarchy for review purposes, but if you can feel your
way around (so to speak) then perhaps that isn't the show-stopper I
thought. I've tried this with the mouse, but the mouse pointer isn't the
same as a static touchscreen.

3. If you can in fact cause items to speak when touched, how doable is
this without seeing the screen? Can you, for instance, reliably find
battery charge/signal strength emulators? If you pop up a dialog and
land in a text area, can you easily and reliably figure out what its
label is?

Of course, usual disclaimers apply, and don't shoot me if this bricks
your phone. :) I'd definitely be interested in reports of how it
performs on actual devices, though. Maybe my concerns are only valid on
the emulator, who knows?

Thanks a bunch.

On 09/27/2009 01:24 AM, blindfold wrote:

blindfold

unread,
Oct 3, 2009, 10:20:21 AM10/3/09
to TTS-for-Android
Hi Nolan,

I downloaded your APK to my ADP1 running Android 1.6 by typing the
shortened URL tinyurl.com/ydwxm9j as my browser destination.
Installation went fine. Next I went to settings and enabled both
Accessibility and Spiel. However, I got no speech yet, but found out
that I also had to do Settings, Speech synthesis, Install voice data.
Perhaps this is ADP1-specific due to its limited memory, I'm not sure.
Anyway, after installing this 4.3 MB package, which takes a few
minutes, it starts shouting about please insert SIM card, emergency
call and so on while the screen reads that it is still "Installing
voice data on SD card". This seemed to take forever, while I got the
warning that phone storage space was getting low, but in the mean-time
the notifications read that the SpeechSynthesis Data Installer was
successfully installed. After about ten minutes or so I got tired of
the screen still visually reading "Installing voice data on SD card"
and pressed the home button. Things seemed to work at least partially
with speech now, and I could navigate my contacts list with the
entries spoken. Sounds quite good so far, but I feel a bit
uncomfortable about the possibly unfinished "Installing voice data on
SD card" part although that most likely has nothing to do with your
screen reader work and everything with the SVOX installation. I just
ran Twidroid, and there too I got speech for the UI elements, but not
for instance the labels alongside the edit boxes. Still, it looks like
you made great progress. Congratulations!!

Next I ran my own fully accessible app, The vOICe for Android, and as
expected I now got "double speech": every entry in its menu was spoken
by both a male and a female voice. The speech in The vOICe is still
based on the "old" Android 1.5 TTS for Android library, so it is not
so strange that we now get two TTS engines talking at the same time. I
have not yet started work on the Android 1.6 TTS features that should
let me drop most of my own accessibility code and avoid this double
speech output.

Peter Meijer


The vOICe for Android
http://www.seeingwithsound.com/android.htm


Nolan Darilek

unread,
Oct 3, 2009, 10:38:42 AM10/3/09
to tts-for...@googlegroups.com
Yay, glad it worked out for you. Comments inline:

On 10/03/2009 09:20 AM, blindfold wrote:
> After about ten minutes or so I got tired of
> the screen still visually reading "Installing voice data on SD card"
> and pressed the home button. Things seemed to work at least partially
> with speech now, and I could navigate my contacts list with the
> entries spoken. Sounds quite good so far, but I feel a bit
> uncomfortable about the possibly unfinished "Installing voice data on
> SD card" part although that most likely has nothing to do with your
> screen reader work and everything with the SVOX installation.

Yeah, I don't actually trigger speech data installation. Ideally it'd be
nice to bundle that with Spiel such that one install rules them all and
the process is somewhat more accessible, but I'm not immediately sure
how to do that.

> I just
> ran Twidroid, and there too I got speech for the UI elements, but not
> for instance the labels alongside the edit boxes.

Yes, this appears to be an API limitation. I get a stream of events, but
can't see a way to get anything more significant. So, for instance, I
might get an event dumping the screen content as an array of Strings,
then events containing the class and content of UI elements that get
focus, but I can't locate the element in the overall hierarchy to make a
guess at its label. This is where my questions about touchscreen access
come in. If you run your finger over non-keyboard-focusable elements,
are they spoken? If labels of text input elements aren't spoken when
focused via the keyboard, is it then possible to find their labels via
the touchscreen and have them read? Not the most ideal, I know, but it'd
at least be something until the API gets more fleshed out in Donut+X.

> Still, it looks like
> you made great progress. Congratulations!!
>
>

Thanks, good to hear. :)

> Next I ran my own fully accessible app, The vOICe for Android, and as
> expected I now got "double speech": every entry in its menu was spoken
> by both a male and a female voice. The speech in The vOICe is still
> based on the "old" Android 1.5 TTS for Android library, so it is not
> so strange that we now get two TTS engines talking at the same time. I
> have not yet started work on the Android 1.6 TTS features that should
> let me drop most of my own accessibility code and avoid this double
> speech output.
>
>

Sweet, would be good to hear from actual app developers about what
functionality I should work on next. :)

blindfold

unread,
Oct 3, 2009, 11:38:48 AM10/3/09
to TTS-for-Android
Hi Nolan,

> If you run your finger over non-keyboard-focusable elements, are they spoken?

Well, on the touch screen pressing check boxes in a menu list gets
spoken, but without their corresponding text label. Only when I move
over the menu list using the joystick are all its items spoken, and
not when I press a list entry. For instance, when I start the built-in
camera app, next press the physical menu key and press the settings
button, your screen reader says "Settings", and I get a menu list.
Pressing a list element on the screen is not spoken, and only any
resulting dialog is spoken. When pressing the "Select video quality"
entry, this string is not spoken, and one hears "Video quality", being
the title of the resulting dialog. If however I navigate the menu list
with the joystick, then I do hear "Select video quality" among the
various other entries.

> would be good to hear from actual app developers about what functionality I should work on next.

I'm not blind so I'm not qualified to judge screen reader
functionality in any detail, but I noticed that your screen reader
only says "button" in case of radio buttons, so I think it would be
useful to refine that. In fact you may find it useful to try the
"double speech" with my app to notice the differences in how dialog
elements are spoken since The vOICe for Android does say "radio
button" where applicable. However, this is a just a detail, and
overall I am very impressed with how far you already got. I also
informed the seeingwithsound user group today, and one question from a
blind member there was "this is interesting. I wonder what the cost
will eventually be". What are your plans that I can report back?

The hardest part is perhaps to deal with proximity in case say an edit
box is positioned near a (visually) corresponding text label on the
screen but without a direct programmatic connection. User interface
elements can be far apart in the GUI element tree but located close
together on the screen such that sighted users will visually link them
together, as was probably intended by the application programmer. One
would then like to have the text label spoken along with the nearby
edit box and its content, but I do not know if the screen coordinates
as needed for doing this can be accessed within the new Android
accessibility framework. If it is not possible at present, then it
would be a strong suggestion to the Google Android developers to add
this information to allow for more intelligent screen reader
functionality. (This must be "intelligent" because one must inevitably
make educated/heuristic guesses about which elements at what screen
separations should be linked together, and which elements should not
be linked in screen reading.)

Regards,

Peter

Nolan Darilek

unread,
Oct 3, 2009, 6:00:07 PM10/3/09
to tts-for...@googlegroups.com
On 10/03/2009 10:38 AM, blindfold wrote:
> Hi Nolan,

>
>
> Well, on the touch screen pressing check boxes in a menu list gets
> spoken, but without their corresponding text label. Only when I move
> over the menu list using the joystick are all its items spoken, and
> not when I press a list entry. For instance, when I start the built-in
> camera app, next press the physical menu key and press the settings
> button, your screen reader says "Settings", and I get a menu list.
> Pressing a list element on the screen is not spoken, and only any
> resulting dialog is spoken. When pressing the "Select video quality"
> entry, this string is not spoken, and one hears "Video quality", being
> the title of the resulting dialog. If however I navigate the menu list
> with the joystick, then I do hear "Select video quality" among the
> various other entries.
>
>

OK, so just to be clear, the default UI doesn't support actually swiping
fingers across the screen, as is done with the iPhone? The only
interaction type is actually clicking with finger presses? Damn, that's
going to make a lot of things difficult or impossible just now.


> I'm not blind so I'm not qualified to judge screen reader
> functionality in any detail, but I noticed that your screen reader
> only says "button" in case of radio buttons, so I think it would be
> useful to refine that. In fact you may find it useful to try the
> "double speech" with my app to notice the differences in how dialog
> elements are spoken since The vOICe for Android does say "radio
> button" where applicable. However, this is a just a detail, and
>

Good point. Currently there's some very hacky code that just matches the
event's originating class against the string "Button," mainly meant to
get things up and running quickly, but it's simple enough to add in
checks for RadioButton and the like before that one triggers.

> overall I am very impressed with how far you already got. I also
> informed the seeingwithsound user group today, and one question from a
> blind member there was "this is interesting. I wonder what the cost
> will eventually be". What are your plans that I can report back?
>
>

Right now it's open source under the MIT license (see
http://gitorious.org/spiel for the code.) No build instructions
currently, but I'll throw together a README in the next few days.

It will always be open source, though license is still debatable. Maybe
MIT, maybe GPL, not sure yet. So anyone willing to build it themselves
has it for free. If it ever becomes usable, I'll probably put it on the
Android Market for $5-$10, basically just to get a few beers for the
effort. :) No intent to actually make a killing on this, since the "AT
market as highway robbery" model isn't one I find particularly
appealing. :) Then again, I'm not sure the Android Market is accessible
just now, and if it uses the browser then it definitely isn't. I really
want to make this usable since I want an Android phone quite badly, but
as was stated earlier in this thread, the current API is very basic.

> The hardest part is perhaps to deal with proximity in case say an edit
> box is positioned near a (visually) corresponding text label on the
> screen but without a direct programmatic connection. User interface
> elements can be far apart in the GUI element tree but located close
> together on the screen such that sighted users will visually link them
> together, as was probably intended by the application programmer. One
> would then like to have the text label spoken along with the nearby
> edit box and its content, but I do not know if the screen coordinates
> as needed for doing this can be accessed within the new Android
> accessibility framework. If it is not possible at present, then it
> would be a strong suggestion to the Google Android developers to add
> this information to allow for more intelligent screen reader
> functionality. (This must be "intelligent" because one must inevitably
> make educated/heuristic guesses about which elements at what screen
> separations should be linked together, and which elements should not
> be linked in screen reading.)
>

Agreed, and based on what I've seen, I'm not sure this is possible at
the moment. I think I've gone as far as I can using the current API, and
thus far I've not gotten any answers to any of my inquiries on these
matters. Hopefully Donut+X will give me more to work with. In the
meantime, I'm filing Android issues based on the limitations I'm
encountering and hoping a few months will bring positive change. Based
on what I've seen on this list, I'm hopeful in that regard.

blindfold

unread,
Oct 4, 2009, 7:10:47 AM10/4/09
to TTS-for-Android
Hi Nolan,

> OK, so just to be clear, the default UI doesn't support actually
> swiping fingers across the screen, as is done with the iPhone?
> The only interaction type is actually clicking with finger presses?
> Damn, that's going to make a lot of things difficult or impossible
> just now.

I don't see a big issue here yet. In my experience, the main common
use of a swipe over the touch screen of Android phones is perhaps to
quickly scroll through a long menu list. However, one can
alternatively use the joystick or trackball to do the same, and the
latter comes with speech using your screen reader, so that's fine. It
is a bit like on the PC where you normally would not be using a mouse
as a screen reader user but would instead use keyboard shortcuts like
the tab key, arrow keys and so on to navigate through menus, dialogs,
drop down lists and what have you.

Other common uses of the swipe seem mostly for moving over large
bitmaps or equivalent, like in scrolling a map in Google Maps, or when
selecting the screen viewing slice in the browser window, but this
seems less relevant to screen reader use and much the same more
fundamental accessibility limitations apply that you have when using
Google Maps on the PC.

Peter


The vOICe for Android
http://www.seeingwithsound.com/android.htm


> Right now it's open source under the MIT license (seehttp://gitorious.org/spielfor  the code.) No build instructions

blindfold

unread,
Oct 9, 2009, 5:31:19 AM10/9/09
to TTS-for-Android
Reply all
Reply to author
Forward
0 new messages