#text entry
actually does anyone have any tips or product recommendations for chronic joint pain in the hands? i’m a compsci major and not being able to type well REALLY sucks
I myself don’t have too much trouble with typing so I usually just suffer through it when it’s bad, but I think @chronicallypainful might know better than I do, she keeps a list of resources and is very well-versed in computers and assistive tech
Yep, I have dealt with this problem. It’s actually one of the main reasons that I didn’t major in computer science. (I got a BS in mathematics. Now a graduate student.) But, I have much better tools now than I did when I was starting undergrad!
You are welcome to poke around this blog or my assistive tech blog @awesomeassistivetechnology, but it is probably easier for both of us to just talk (on Skype or Google hangouts or whatever). And there’s also a fair bit of knowledge in my head that has never made it onto either blog. So, yeah, send me a message if you’d like to chat. One of my goals for the summer is actually to strengthen my social support networks, so it would be cool to “meet” you. :-)
I’m a professional programmer and completely use Dragon naturally speaking to dictate code because of chronic pain. There‘s a really cool open source software community around it. You need to install natlink, python, dragonfly, and my favorite tool set is called caster. There’s a YouTube channel called castervoice. It works pretty well, all things considered.
Very cool. Thanks for sharing! I currently use Dragon for emails/blogging/etc and LaTeX. And I’ve been using eye tracking (low cost eye tracker marketed to gamers) with an open source keyboard called OptiKey for writing code and working on the command line. I’ve experimented with natlink and vocola in the past, but never reached the point of finding them really useful.
One of my biggest challenges with Dragon is that it doesn’t cooperate with most code editors. May I ask what what editor you use?
Nope, not kidding. I have actually been writing computer programs with my eyes (and a few voice commands thrown in here and there).
More specifically, I’m using an eye tracking device called the Tobii EyeX. It’s a consumer-level eye tracker which mounts to the bottom of my computer monitor and connects to my computer via USB 3.0. (I’ve written about this type of eye tracking device before, and you can also look at Tobii’s website.)
Okay, so how?…
Good question! And I’m going to answer it with the rest of this post. I’m going to write this in the general style of a how-to guide, in case any of my readers actually want to copy my set up. (To my knowledge, no one else has documented how to do this.) However, even if you have no desire to write Python with your eyes, you might still want to skim the rest of this. Several of the tools that I’m using have potentially much broader applications.
Hardware requirements:
-Tobii EyeX sensor bar (link, < $150)
-external monitor, preferably at least 20 inches but no more than 24 inches (You probably could make this work with a laptop, but having a larger screen area makes the eye tracker much easier to use. 24 inches is the size limit for the EyeX sensor.)
-Modern computer with USB 3.0 support (I have a quad core processor at 3.6 GHz, 8 GB of RAM. I wouldn’t recommend much less than that, unless you want to eliminate the speech commands from the project and use keystrokes instead.)
Software requirements:
-Windows 7 64-bit (should work on Windows 8 or 10, as well. But you may have to make some adjustments.)
-EyeX drivers and associated software (included with the device)
-Dasher (free and open source)
-FreePIE(free and open source)
-Compiler and editor for the language you wish to program in (for this example, I’ll be using Sage and its browser-based interface)
You will also need some sample code from the language you want to program in (so that Dasher can build an appropriate language model). You can ignore this if you only want to write in English (or another natural language).
Got all that? Good, let’s get started.
1. Install the EyeX drivers and mount the sensor according to the manufacturer’s instructions. Run the configuration to get EyeX to work properly with your monitor, and calibrate it to recognize your eyes. All of this is standard for any user of the EyeX, and you should be able to follow instructions from the manufacturer. When you’re finished, you can test your calibration using the “test calibration” option in the EyeX software.
2. Download and install Dasher. We will make some configuration changes to Dasher in the next step, but for now just take a minute to familiarize yourself with Dasher if you haven’t used it before. With the default settings, you can click the mouse inside of Dasher’s window to start writing, and drag the mouse towards the letters you want in order to write. Your text will appear in the box at the top. For more details, see Dasher’s website. (Obviously, skip this experimentation/exploration step if you aren’t physically able to use the mouse. You can experiment with Dasher once we get it working with the eye tracker. I just included this step so that, as much as possible, new users can get used to one new technology at a time.)
3. (Skip this step if you just want to write in English, not computer code.) Close Dasher, and locate its program files on your hard drive. On Windows 7, I have a Dasher folder in Program Files (x86). Inside of that is another folder for my specific version, in this case Dasher 4.11. In your Dasher folder, find the folder labeled system.rc. We are going to add two new plaintext files to this folder. First, an alphabet file. This defines the all of the valid symbols for your language. I would recommend starting with an existing file, so make a copy of alphabet.englishC.xml and put it in the same folder. Name your copy alphabet.<yourlanguage>.xml, and open the file in a plain text editor. Make the following changes :
-Change line 5 so that the text in double quotes is the name of your language (this will be shown in that the user interface).
-Change line 9 by replacing ‘training_english_GB.txt’ with the filename for your training text. (I used 'training_sage.txt’)
-Remove the section labeled “Combining accents” (lines 13 through 27)
-(optional and language-dependent) I additionally removed line 127 (the tab character) because my editor does auto-formatting with tab characters, and unexpected results were generated if Dasher inserted a tab character and then attempted to delete that character with the backspace key (which it regularly does if you take a “shortcut” to the character you want through another box). I decided that it was simpler to leave the autoformatting on in my editor (I really didn’t want to manually insert six or seven tab characters before a line of code, as can happen in Python!) and insert my tabs manually with voice commands. You will have to decide how to handle these sorts of formatting and autoformatting issues yourself, based on the language and editor that you choose.
It is also at this point that you can reorder or regroup any of the characters if you want, in a way that makes sense for the language you are writing.
Save your changes, and now you have an alphabet file. Now, let’s specify the training text. Create a plain text file titled training_<yourlanguage>.txt (or whatever you called it in the alphabet file). Paste all of your sample code into that file. (More is better! But Dasher will also learn from what you write over time.) Save your training text file in the system.rc folder, just like the alphabet file.
Now, open Dasher and make sure that you can access your new language. Click on the Prefs button, and look in the alphabet selection box. You should be able to locate the language that you just specified. Click on it, click okay, and try writing in your programming language. If everything is working, proceed to the next step. Otherwise, go back and check that you got all of the syntax and file names/locations correct. (A few troubleshooting details: If it works, but Dasher warns you that you have no training text, then your alphabet file is correct but you need to check the training text file (or file specified for training in the alphabet file). If it works, but the predictions seem bad, then you probably need more training text. If it works, but Dasher takes too long to load, try deleting some of your training text.)
4. Okay, now let’s configure the Dasher user interface. Click on the Prefs button. Under alphabet selection, make sure your desired language is selected. On the control tab, select 'Eyetracker Mode’ for 'Control Style’. Then, select 'Mouse Input’ for the 'Input Device’. You can select the speed and choose whether or not you want Dasher to automatically adjust its speed. (I like this feature, but it’s up to you.) Finally, choose the method(s) for starting and stopping. Personally, I like the “Start with mouse position – Centre circle” option for with the eye tracker. You might also want to enable the mouse button control and/or spacebar, especially until you get comfortable with the eye tracker (obviously only useful if you have a way to push the spacebar or click the mouse button). I would not recommend using the “stop outside of canvas” option, because it makes the motion feel extremely jerky when the eye tracker briefly leaves the active area. All of that is personal preference, though. There’s no harm in experimenting to see what you like best.
5. More Dasher configuration. In the preferences window, click on the Application tab. Under “Application style:” select “Direct entry”. This means that Dasher will automatically send keystrokes to the currently active window. This lets you write directly into your code editor of choice.
Let’s test again. With Dasher open, open Microsoft Notepad. Click in Notepad so that your cursor is in the text entry field. Then go to Dasher and (without clicking on anything besides the canvas – you don’t want to change focus) start writing with the mouse. Your text should appear in Notepad. Once that’s working, give it a try with your desired code editor.
6. Download and install FreePIE.
7. Let’s actually use this eyetracker! At the end of this post, I have pasted the code for a simple mouse control script. Take this script and paste it into a new script in FreePIE. Click on script menu and then run script (or press F5). (Tumblr doesn’t support any sort of code environment, so, if you have trouble getting the script to work, let me know and I can email you the actual file.) Then press the Z key to start the eye-controlled mouse movements. If everything is working correctly the mouse cursor should now follow your gaze around the screen. I’ve also set up this script so that you can start or stop the eye mouse using speech. If you have a decent microphone (preferably headset), make sure that that device is selected as your default recording device in Windows. Then, with the script running, you can say “start eye mouse” or “stop eye mouse” to start or stop the mouse movement. (If the computer has a hard time recognizing your voice, you can improve this by doing some of the training in Windows speech recognition. You can find it in the control panel.)
8. Now all that’s left to do is chain all the pieces together. Get FreePIE open with your mouse script running. Make sure that Dasher is running, and also open your code editor. Make sure that the focus is in the text entry field of the code editor, press the or use speech to start the eye mouse, and finally start Dasher (by whatever you selected in Dasher’s configuration). You should now be able to enter code with only your eyes.
9. (Optional) Depending on your needs, you may want to add some additional commands to the FreePIE script. For example, I have written some code that lets me enter tab characters by voice, and also navigate with in text by voice (e.g. “go to end of line”, “go to beginning of line”, “go up line”, “go down line”, etc.). Each of these is just a couple of lines in FreePIE. Feel free to reach out to me if you need some help with adding these sorts of additional commands.
That’s a lot of text, so let’s finish off with a screenshot. Here’s what the whole thing looks like when it’s up and running. On the left-hand side of the screen you see Dasher. On the right-hand side is Firefox showing Sage. FreePIE is running but minimized.
[Image description: the image shows a screenshot of a Windows 7 desktop with two visible applications. On the left-hand side of the screen is Dasher. The majority of that window is filled with a collection of different colored squares and typable characters. There is a cross in the center of the window, and a red line shows where the user is currently pointing. On the right-hand side of the screen is a Firefox web browser window. The active tab is labeled Palindromes, and most of the window is occupied by text fields showing Sage code and output.]
The script for FreePIE:
#Use Z or speech to toggle on/off
import ctypes
import math
def update():
global prevX
global prevY
#Settings
smoothingConstant = 0.2 #between 0 and 1; 0 means no cursor movement, 1 means use raw data
rawxcord = tobiiEyeX.gazePointInPixelsX
rawycord = tobiiEyeX.gazePointInPixelsY
diagnostics.watch(rawxcord)
diagnostics.watch(rawycord)
xcord = prevX + (rawxcord - prevX) * smoothingConstant
ycord = prevY + (rawycord - prevY) * smoothingConstant
if enabled:
ctypes.windll.user32.SetCursorPos(int(xcord),int(ycord))
global enabled
#enabled = False
prevX = xcord
prevY = ycord
if starting:
global enabled
global prevX
global prevY
enabled = False
prevX = 0
prevY = 0
tobiiEyeX.update += update
toggle = keyboard.getPressed(Key.Z)
start = speech.said(“start eye mouse”)
stop = speech.said(“stop eye mouse”)
if toggle:
global enabled
enabled = not enabled
if start:
global enabled
enabled = True
if stop:
global enabled
enabled = False
Just realised that Google docs has a voice recognition program, under the Tools menu.
It’s a lot better than the Dragon naturally speaking software, although of course it’s limited to word processing
I thought I’d mention this for anyone who has trouble using their hands.
- Mod S
Yeah, it does! It’s actually relatively new.
I would hesitate to say that it’s better than Dragon overall, but it certainly has strengths that Dragon doesn’t, especially for working in noisy environments or with low-quality microphones. Dragon is still better for entirely hands-free work, and has more customization options if you have non-typical usage needs, though. Of course, the Google docs tool is free, so that’s a major plus!
If anyone’s interested in some more information, I wrote a short post about the new Google docs feature here. And I have an old post covering all the basic info about speech recognition / dictation for computer input.
Within the past year, multiple companies have released eyetrackers costing hundreds, instead of (tens of) thousands, of dollars. These new devices are aimed at gaming and multimedia applications, but they are still amazingly useful as assistive technology. The two main competitors at this point seem to be Eye Tribe andTobii EyeX.
I purchased an EyeX unit this March, and it has significantly expanded what I can do with my computer. I will write later about the specifics of my setup, but I want to today just survey some of the development happening on the EyeX and other low-cost eye tracking devices (much software works across multiple devices) which is relevant to disability. (There is also significant gaming development on the platform, but it is largely using multiple input devices and would not be accessible to many physically disabled people. Plus,Tobii will tell you all about that.)
To be clear, I’m saying that these devices have great accessibility potential, not that they are currently ready made computer access solutions. At present, using these devices as an accessibility tool requires at least moderate technical skills.
Okay, without further ado, here are some software projects I find interesting:
- OptiKey is an on-screen keyboard and mouse emulator specifically designed for eye tracking input. Free and open source. Windows only.
- Project Iris uses eye tracking to control the mouse. It also allows you to define active screen regions and triggers actions when you look at them. Many potential uses for accessible gaming. Paid. Windows only.
- Gaze Speaker is a self contained environment for speech generation by eye movements. Also includes other tools like an email client, a simple web browser, and a file browser, all designed to be accessed by eye alone. Free and open source.
- FreePIE lets you write Python scripts that take input from many, many different devices, including the EyeX. You can use that input to trigger just about any action you can imagine, provided you can figure out how to write the code. Probably best for geeks, but extremely powerful for meeting custom needs. Free and open source.
Voice dictation has finally made its way to Google Docs! You can find it in the tools menu under “voice typing.” And hereis the official documentation from Google.
I’m using it to write this right now. The accuracy is quite good, certainly better than when I first experimented with Google’s technology a few years ago. I’m guessing it’s comparable to Dragon at this point, at least for non-specialized vocabulary. (Though it does seem to make different kinds of mistakes, which I’m guessing is a result of being trained on different types of data.)
However, it lacks full voice control of the user interface. Especially notable is the fact that you cannot make corrections to recognized text by voice. (Though alternate recognitions are available from an on-screen menu, you have to click with the mouse.)
For me, these user interface limitations mean that Dragon will continue to be my tool of choice for the foreseeable future. However, I will certainly keep this new tool in mind for when I am away from my desktop computer.
And I LOVE seeing competition in the speech recognition arena! We now have Nuance, Google, & Microsoft working on these problems, plus a free/open-source project from Carnegie Melon. Competition drives innovation, and I can’t wait to see some more innovation!
