Article: Talking TMAP: Automated Generation of Audio-Tactile Maps Using Smith-Kettlewell’s TMAP Software Submitted to: British Journal of Visual Impairment Joshua A. Miele, Ph.D. Research Fellow, The Smith-Kettlewell Eye Research Institute San Francisco, CA USA Email: jam@ski.org Steven Landau, M.Arch. President, Touch Graphics, Inc. New York, NY USA Email: sl@touchgraphics.com Deborah Gilden, Ph.D. Senior Scientist, The Smith-Kettlewell Eye Research Institute San Francisco, CA USA Email: debby@ski.org January 28, 2006 Introduction The Smith-Kettlewell Eye Research Institute’s Tactile Maps Automated Production (TMAP) project addresses the blind or severely visually impaired traveler’s need for geographical information about unfamiliar environments (Miele, 2004). To help meet this need, a prototype web site for rapidly producing high-quality, low-cost tactile street maps has been developed. It brings together online Geographic Information Systems (GIS), accessible web design practices, tactile graphics design principles, and modern tactile hardcopy production methods to enable a totally blind individual to independently produce customized tactile street maps (Miele & Marston, 2005). By incorporating touch tablet technology in the form of the Talking Tactile Tablet (TTT) from Touch Graphics, Inc. (Landau & Gourgey, 2001), the scope of the project is now being expanded to include the automated production of audio-enabled tactile street maps. The necessity for tactile street maps for orientation and mobility Sighted people take for granted ready access to maps, signs and other visual aids for navigation. In planning a trip to an unfamiliar town, or finding one’s way around upon arrival, using maps is routine, and most sighted people could not imagine a world without them. For blind people, on the other hand, access to a tactile street map that is detailed enough to use for travel within any particular area is an extreme rarity. The dearth of tactile maps has far reaching implications, well beyond the orientation and mobility domain (Golledge, 1993). The expense and difficulty of producing tactile graphic materials in general means that young blind children have inadequate exposure to them, thereby precluding optimal development of spatial, graphical and map reading skills. Tactile street maps are beneficial to blind pedestrians across a wide range of age groups from elementary school-age, well into late adulthood (Espinosa & Ochaíta, 1998). Although some early studies questioned the cognitive ability of blind individuals to interpret maps, most recent results indicate that with some training, they can and do make use of maps to inform their internal cognitive representation of space (Uttal, 2000). Compared to control subjects, blind and visually impaired people who are able to feel tactile diagrams, and who have had the opportunity to study a tactile map of an area of interest, show improved ability to independently navigate within that area (Ungar, Blades, & Spencer, 1993). Tactile maps can also be an extremely effective tool for representing spatial information for the orientation and mobility (O&M) student (Bentzen, 1997). Many O&M instructors devote a significant portion of their lesson preparation time to the creation of detailed tactile maps to be used only once or twice by the student. An automated technique for producing highly-detailed, “one-off” maps would be of great benefit to these instructors and their students, freeing the instructor’s time for actual O & M instruction, instead of the arduous task of tactile cartography. The TMAP software uses a geographic database to build its maps. Currently, the data being used are the US Census Bureau’s Topologically Integrated Geographic Encoding and Referencing System (TIGER®) line data. This data set includes the names and locations of virtually every street in the United States. As the TMAP project expands its scope, a commercial data set will be incorporated that will have richer and more accurate data, as well as geographic data for many other countries. Regardless of the source, street locations are stored as vectorized latitude and longitude points, rather than as images. This makes it possible to construct tactile maps from scratch, rather than trying to transform existing “visual” maps to tactile ones. The TMAP software generates tactile maps according to a set of labeling, street density, line style, and scaling criteria intended to maximize tactile map readability. The maps can be printed out on the user’s own Braille embosser at minimal cost, or embossed remotely and mailed to the user. Figure 1: A visual representation of a sample TMAP A visual representation of a sample tactile map generated by prototype TMAP software appears in Figure 1. It was generated simply by specifying the street address, along with some scale and labeling parameters. Map features include lines representing streets, abbreviated Braille street labels, a scale indicator in the upper-right corner, a map title at the top of the page, and a donut-shaped marker indicating the location of the requested street address or intersection. This illustration happens to show the area of San Francisco in the immediate neighborhood of The Smith-Kettlewell Eye Research Institute, but could just as easily show any other neighborhood. A tactile map can readily reveal the shapes of intersections, the direction a street curves, whether a street dead-ends, and many other geographical features which are evident visually but are extremely difficult for an independent, visually impaired traveler to distinguish without previous knowledge. A tactile map also allows the traveler to inspect an unfamiliar area in advance and to plan a route to the desired destination, saving time and frustration, and avoiding dangerous situations such as attempting a street crossing at an unusually-configured intersection. Technical description: The Tactile Maps Automated Production (TMAP) model Web-based map generation The TMAP model consists of a user interface, the TMAP Engine, and a map production step. The user interface will include both a web site and a telephone interface, although to date only the web site has been initiated. The map files – digital graphics files specially prepared for production on a Braille embosser or other device – are produced by the TMAP Engine. Among other functions, the TMAP Engine incorporates algorithms for applying the individual user’s preferences, finds space on the map for abbreviated Braille labels, and determines if the user-specified scale is reasonable for the density of geographical features. Once the digital map file has been generated, the user can produce the physical map either by downloading the file from the web interface and sending it to a local Braille embosser, or by requesting that the digital file be sent to a third party capable of producing the tactile map. If produced by a third party, the map and its associated legend are sent to the user via regular mail subsequent to production. Figure 2 illustrates the steps a consumer would go through when using TMAP. Figure 2: The steps for getting a map via TMAP One of the most difficult aspects of automating the generation of tactile maps is placing Braille labels to identify particular map features. Visual maps often use small print and flexible text orientation to associate a large number of labels with their referents. Printed text labels can even be curved to fit the contour of a road or other map feature, thus associating the two by shape as well as proximity. Nothing of this sort is possible with Braille. Potential confusion from mixing Braille with tactile graphics imposes significant limitations on how map features can be labeled. Also, Braille takes up much more room than print and cannot be resized to fit an available space: there is no such thing as “fine print” in Braille. TMAP uses an algorithm to generate a unique set of abbreviations for a given list of feature names. The abbreviated labels are placed on the tactile map adjacent to the associated feature, and the abbreviation and full feature name are included in a legend provided on a separate page. The algorithm allows the user to select the number of characters to be used in the label. For example, Fillmore Street might be abbreviated “FL,” “FLM,” or “FLMR,” depending on whether the user chooses to use two, three, or four characters per label. The advantage of including more characters is that it makes the associated feature names easier to understand and remember. The advantage of fewer characters per label is that it leaves more room available for map features and other labels. The TMAP researchers have investigated several labeling schemes, but have focused primarily on a technique that places feature labels around the perimeter of the map. In this approach each street label is adjacent to the point where the street intersects the edge of the map. The advantage is that the user knows that all of the labels are around the perimeter of the map. This significantly reduces map clutter and label ambiguity. The disadvantage is that streets that do not intersect the edge of the map are not labeled. Also, the limited space at the edge of the map leads to labeling conflicts, making it necessary to omit labels for some streets. One solution to the problem of effective TMAP annotation is to add interactive audio tags, an approach that will be discussed in detail later in this article. Tactile output The maps provided by the TMAP Project are primarily designed to be produced using Braille embossers. The raised dots can be arrayed in tight formations to produce tactile figures such as lines, curves, and shaded areas. This technology has a solid and proven track record, and has been used effectively for producing tactile graphics for over 30 years. There are large numbers of Braille embossers currently in use in homes, offices, schools, libraries, and other public and private institutions. This means that with the availability of the TMAP tools, many visually impaired users will be capable of producing their own tactile maps. Tactile Graphics File Formats and Scalable Vector Graphics (SVG) In addition to supporting file formats for all graphics-capable Braille embossers, TMAP can provide tactile maps for any number of additional production techniques by using a universal graphics file format known as Scalable Vector Graphics (SVG). A number of alternatives to embossed tactile output exist, the most common of these being micro capsule (or “swell”) paper (Pike, 1992). A significant amount of research has also been conducted in the area of using inkjet technology for producing raised lines and textures (McCallum, 2003). In order to avoid the necessity of providing an unspecified number of tactile graphics file formats as new technologies emerge, the TMAP software can produce output in the form of Scalable Vector Graphics (SVG) files. This file format is based on the extensible markup language (XML) as specified by the World Wide Web Consortium (W3C), and will allow third-party researchers and developers to render tactile graphics from a standardized data format. Using the SVG file format, each geographic feature can have individual attributes assigned to it, such as title and description tags. Furthermore, objects can be grouped together and assigned various collective attributes. For example, many streets are composed of multiple segments, or “blocks.” These segments are represented as individual elements in the SVG file, but can be grouped together and given shared attributes such as the street name or description. The SVG format is extremely flexible and will allow other features such as points of interest, map regions, and bodies of water to be added in the future. By providing this kind of detailed structural and semantic information about the maps generated by TMAP, the number of uses for the TMAP data is greatly expanded. The Talking Tactile Tablet (TTT) and TMAP One way to get around the difficulties of Braille-based labeling of tactile maps is through interactive audio tagging. To test this approach, the TMAP developers at The Smith-Kettlewell Eye Research Institute in San Francisco have teamed with Touch Graphics, Inc., a New York City company that has developed the Talking Tactile Tablet, nicknamed TTT or T3. The TTT is a portable, rugged and inexpensive computer peripheral device that acts as a “viewer” for tactile graphic materials. Users place one of a collection of raised-line and textured overlay sheets on the device which measures about 15 inches wide, 12 inches deep, and one inch thick (see Figure 3, a photograph of the device with a sample overlay sheet mounted). Overlays are held in position against a touch-sensitive surface by a heavy hinged frame that the user can open and close. The TTT can detect a finger press through the tactile overlay sheet and transmits that position to a PC via a USB connection. The computer runs a program that compares the positions of each finger press on the picture, map or diagram with a list of pre-defined hotspots, and responds appropriately. Figure 3: The Talking Tactile Tablet with the UK Map sheet mounted. Photo credit: Devon Jarvis Many researchers have contributed to the field of audio-enabled haptic exploration and tactile graphics (Gardner & Bulatov, 2001; Krueger & Gilden, 1999; Parkes, 1988). The TTT system adds a Tactile Graphic User Interface (TGUI) consisting of a uniform set of simple tactile symbols arrayed identically around the central workspace on each overlay sheet (see Figure 4, the Japan and Korea map from the National Geographic Talking Tactile Atlas of the World). The standardization of the TGUI makes it possible to construct audio-tactile applications that rival mainstream mouse and video-based multi-media presentations for interactive richness (Landau, Russell, Gourgey, Erin, & Cowan, 2003). An identification routine allows quick and easy access to the digital data associated with each tactile figure. After placing the sheet on the TTT, the user presses three specific points, which appear as short vertical bars along the horizontal ID Strip located just below the top edge of every sheet. Since the position of the short vertical bars is unique for each sheet, the system is able to identify a sheet by comparing these positions to a database of known combinations. Upon completing the identification process, the data for that overlay sheet is loaded. The user can then touch any position on the tactile figure to hear the information associated with that location. When the user wishes to change sheets, he or she simply repeats the process with the ID bar of any new sheet. Figure 4: The Japan and Korea map. Labels show functions of various elements of the Tactile Graphics User Interface, and do not appear on the overlay sheet. The Talking TMAP application Touch Graphics has developed a software application, known as Talking TMAP, that uses a TTT and an SVG file (downloaded from the TMAP web site) to completely automate the process of adding audio annotations to a tactile map. The application, implemented in the Macromedia Director® authoring environment, facilitates interaction between a user and the TTT’s host computer. This section will describe its operation and some design aspects. The Talking TMAP program automatically loads map data that has been generated by the TMAP server. This data arrives at the user’s computer via Internet download, as a single text-based file in the SVG format, permitting the definition of any number of shapes and associated text-based information. Each block of each street shown on the map is described as a series of vertices along a linear path. The program loops through the SVG data, and builds an on-screen version of the map by drawing a thick line linking each of the vertices for a given street. The TTT is seen to the host computer as a pointing device. Thus, when a user places a TMAP on the TTT and then presses tactile lines on its surface, the computer identifies which street is being pressed, just as if a sighted user had clicked a mouse on a visual representation of the map. Making new maps for Talking TMAP is straightforward. Users simply visit the TMAP web site and use the online forms to request a tactile map. During this process, the user specifies that he or she will be viewing the map using the TTT by selecting “TTT” from a number of possible output formats. The user then opens the resulting SVG file with the Talking TMAP Creator program. The SVG information is automatically converted to a format appropriate for the user’s Braille embosser, as well as a computer file that associates objects and attributes with positions on the TTT. The Talking TMAP Creator program sends the map file to the embosser, which produces the finished, ready-to-use tactile overlay. The overlay includes the TGUI elements and the unique ID bar that associates the overlay sheet with the digital file containing the map specification. At the simplest level, Talking TMAP allows the user to freely explore a map of a neighborhood by pressing on streets to hear their names spoken via the computer’s speech synthesizer, but a number of other types of geographical content and software functionality are also available to the user. A layering strategy permits access to additional classes of information so that they are easily accessed by the advanced user, but in a controlled way, to ensure that a less-sophisticated user is not overwhelmed. When a street is pressed and quickly released on the tactile map, just the name of the street is spoken. But by tapping repeatedly on the same street, other layers are revealed. In the current version of the application, the second tap causes the system to speak the address ranges for the right and left sides of that particular block, the third tap announces the length of the block, and the fourth tap causes the system to spell the street name. As with all applications for the TTT, Talking TMAP includes a Main Menu of special functions that are accessed whenever the plus sign shape in the upper right corner of the overlay sheet is pressed (Figure 5 shows a program logic flow diagram). Users navigate among three Main Menu choices by pressing the up or down arrows to move incrementally through a list, then press the circle button to choose one. The Main Menu options are: * Index. All map readers need a way of searching a list of every place shown on a map, as well as an easy method for finding the position of any item in the list. With Talking TMAP, the user selects the Index tool, and then scrolls through an alphabetical list to find a feature of interest by repeatedly pressing arrow buttons, followed by the circle button upon hearing the name of the desired feature. Then, he or she is guided to the requested destination on the map through a process of verbal coaching. To accomplish this the user is instructed to touch the map anywhere and then follow the guiding instructions which incrementally lead to the target. Two advanced index options are available for more sophisticated users. The first of these guides the user to an intersection of two streets, first by selecting a street from the index, and then by selecting another street from a list of only those streets that cross the first one. The second advanced option allows the user to select a street from a list, and then enter an address using the tactile number keypad on the overlay sheet. If the address appears on the map, the user’s finger is then led to that location. * Distance Calculator. Because maps are produced to a known scale, it is possible to determine the linear distance between any sequence of way points. This is useful in trip planning, and in Orientation and Mobility training, where students are taught to think about distances traveled along each leg of a route. When the distance calculator is active, pressing a point on the map for more than one second causes a waypoint to be added to the route. A tone sounds and the user is prompted to press the next point along the route. During this process, brief touches of the map continue to announce feature names, permitting easy exploration of the map while building a route. When finished, the user simply presses the circle button of the TGUI to hear total distance along the route, calculated in the preferred units of measure. * Settings. User preferences can be set in three areas: sensitivity, which controls the firmness with which the user must press on the overlay sheet to cause it to respond; units of measure for distance calculations (choices are meters, feet, kilometers and miles); and speech rate for the synthetic voice. The arrows and circle button of the TGUI are used to select desired preferences, and those settings are saved for use in future sessions. Figure 5: A diagram illustrating the Talking TMAP program flow. Research plans Even though this project is at an early stage of development, it is important to gain some insight about how this approach is received by members of the intended target population, how they use it, problems they might encounter, etc. If designs need to be modified, it is much easier and more efficient to make changes earlier rather than later. As part of the Talking TMAP development effort, some simple usability experiments will be conducted to address some of these questions Although the system described above uses synthetic speech output, it is also possible to provide the same information via an interfaced refreshable Braille display. Braille output would expand the potential use of Talking TMAP to deaf-blind individuals who can read Braille – a population greatly in need of more access to information. Therefore, both blind and deaf-blind individuals will be recruited for the pilot study. Subjects Twelve adults, six men and six women, with vision impairment severe enough to preclude the use of print maps, will serve as subjects. Four of the subjects will also have a hearing loss severe enough to require some form of tactile communication, and will be proficient in receiving tactile sign language. For ease of reference, these subjects will be referred to as “deaf-blind.” All 12 subjects will be competent Braille readers and have good orientation and mobility skills. Procedure Each subject will be tested individually. At the start of the sessions, subjects will answer a questionnaire about their visual impairment, braille-reading skills, mobility skills, and experience with tactile maps. Upon completion of the questionnaire, blind subjects will be shown a practice tactile map (previously generated by TMAP) overlaid on a Talking Tactile Tablet. The map will be of a real, but unfamiliar, neighborhood in a different US State. Subjects will be given instructions on how to press different parts of the map to hear relevant speech messages. For this experiment the information will be restricted to the map’s scale, street names, and intersection names. Deaf-blind subjects will be shown the same practice map overlaid on the TTT, but rather than using the speech-generated information, they will receive their information via an interfaced refreshable Braille display. The information on the Braille display will be identical to that of the speech output. After the training period, both groups of subjects will be shown a test map. The test map will also be of an unfamiliar neighborhood in a different state, and of a complexity similar to that of the practice map. Subjects will be asked to determine the map’s scale, locate particular streets, locate particular intersections, find the center of the map, estimate the distance between two intersections “as the crow flies,” trace and describe an efficient route between two intersections, and estimate the distance of that route. As in the practice condition, blind subjects will receive dynamic information about the map via synthetic speech and deaf-blind subjects will receive the same information via a refreshable Braille display. Sessions will be video taped. Data Analysis This initial pilot study is designed to provide insights about how blind and deaf-blind individuals approach the task of gleaning information from raised line maps equipped with speech or refreshable Braille output. Thus our focus will be on acquiring observational data. Once Talking TMAP is more refined, larger population samples will be used to acquire more objective data. Nevertheless, subjects in the current study will be scored on the accuracy of their answers and the time to answer correctly. By observing the subjects and reviewing the video tapes, it will be possible to note how each subject first examines the map, and the exploration strategies he or she uses to answer the questions. Issues of interest include: * How much variation is there in how individuals initially approach a tactile map? For example, is there a tendency to generally start with a “broad stroke” to get a Gestalt overview rather than to start with a series of smaller areas? Is there a part of the map that is generally examined first? * Does initial exploration approach relate to ability to answer the test questions? * Do gender, age of vision loss, past experiences with tactile maps, or concomitant hearing loss play a role in map-reading ability? * Do subjects seem comfortable with speech output and Braille output? * Does moving hands between the tactile map and the Braille display cause problems? Future directions for Talking Tactile Maps The work described here is part of a six-month feasibility study funded under a Small Business Innovation and Research (SBIR) grant from the National Institute on Disability and Rehabilitation Research (NIDRR), a division of the US Department of Education. Depending on outcomes and the availability of future funding, it may lead to an additional 24-month follow-on project. This would allow further development of the Talking TMAP concept to the point where it can be disseminated among the user community. The Phase 2 work would set out the following goals: 1. Add additional classes of information. In the current version, the amount of information available is limited by what can be extracted from the public-domain TIGER® Line data. However, one of the great virtues of the audio-tactile approach is that there is theoretically no limit to the amount of data that can be usefully embedded in a map, as long as the layering strategy described above is used to provide adequate reader control over playback. Some additional classes of information that might be added are: number of lanes of traffic; direction of traffic flow; whether streets are considered major arteries; locations of important buildings and landmarks; and information about non-street cartographic elements such as railroad tracks, coastlines, bodies of water; etc. 2. Expand Talking TMAP to other kinds of maps. The same technology and methods that permit the creation of talking tactile street maps of individual neighborhoods can be extended to include other kinds of map products. Phase 1 concentrates on materials mostly intended for use in way-finding and orientation and mobility training, but in Phase 2 it would be possible to start producing maps showing entire cities and towns, regions, and countries. There are many potential uses for these maps. For example, a person living in New York City might want to examine a map of the entire state to find out about the relative distances between cities, prior to deciding whether to take a new job. Expanding the scope of the TMAP system will require additional study of the kinds of symbols and textures that would be needed to show a wider range of map entities that could be made effectively with current Braille embossers (Rowell & Ungar, 2003). 3. Develop a Talking TMAP production service. Many potential users will not have access to a Braille embosser, and others may not have Internet connections, or will lack the sophistication to place requests for maps on line. It must be recognized that, while some blind and low vision individuals are highly capable computer users, many others lack experience and have difficulty working with various access technologies. For these individuals, a map-request system that uses the telephone, email or postal mail would be desirable. Touch Graphics might establish such a service, allowing the map and associated computer files to be sent to the user via post. Upon its arrival, the user would place the digital medium (such as CD or compact flash card) in his or her computer, place the raised-line map overlay on their TTT, and begin. This distribution model is already in place and familiar to customers (in the US) who order Braille materials and talking books via mail from the National Library Service or Reading for the Blind and Dyslexic. While this solution still requires some computer literacy and access to equipment, it will be easier for those who might have only minimal computer skills. Conclusion The lack of accurate, detailed and accessible street maps has long been a significant impediment to many visually impaired individuals as they strive to lead independent and productive lives. The work discussed here on the TMAP and Talking TMAP systems tries to address this deficiency in practical and effective ways. Based on positive outcomes in upcoming user trials, this work may lead to the introduction of real-life solutions to this persistent problem in the very near term. References Bentzen, B. (1997). Orientation Aids. In B. Blasch, R. Wiener & R. Welsh (Eds.), Foundations of Orientation and Mobility (2nd ed., pp. 284-316). New York: AFB Press. Espinosa, M. A., & Ochaíta, E. (1998). Using tactile maps to improve the practical spatial knowledge of adults who are blind. Journal of Visual Impairment & Blindness, 92(5), 338-345. Gardner, J., & Bulatov, V. (2001, March). Smart Figures, SVG, and Accessible Web Graphics. Paper presented at the CSUN International Conference on Technology and Persons with Disabilities, Los Angeles, CA. Golledge, R. G. (1993). Geography and the disabled: A survey with special reference to vision impaired and blind populations. Transactions of the Institute of British Geographers, 18, 63-85. Krueger, M. W., & Gilden, D. (1999). KnowWare ™: virtual reality maps for blind people. In J. D. Westwood (Ed.), Medicine meets virtual reality (pp. 191-197). Amsterdam: IOS Press. Landau, S., & Gourgey, K. (2001). Development of a Talking Tactile Tablet. Information Technology and Disabilities, VII(1). Landau, S., Russell, M., Gourgey, K., Erin, J., & Cowan, J. (2003). Use of the Talking Tactile Tablet in Mathematics Testing. Journal of Visual Impairment & Blindness, 97(2), 85-96. McCallum, D. R., J.; Ungar, S. (2003). Producing Tactile Maps Using New Inkjet Technology: an Introduction. The Cartographic Journal, 40(3), 294-298. Miele, J. (2004, March). Tactile Map Automated Production (TMAP): Using GIS Data to Generate Braille Maps. Paper presented at the CSUN International Conference on Technology and Persons with Disabilities, Los Angeles, CA. Miele, J., & Marston, J. (2005, April). "Tactile Map Automated Production (TMAP): On-Demand Accessible Street Maps for Blind and Visually Impaired Travelers. Paper presented at the Annual Meeting of the American Association of Geographers, Denver, CO. Parkes, D. (1988, June). NOMAD: An audio-tactile tool for the acquisition, use, and management of spatially distributed information by partially sighted and blind people. Paper presented at the Second International Conference on Maps and Graphics for Visually Disabled People, Nottingham. Pike, E., Blades, M., Spencer, C. (1992). A comparison of two types of tactile maps for blind children. Cartographica, 29(3 and 4), 83-88. Rowell, J., & Ungar, S. (2003). A Taxonomy for Tactile Symbols: Creating a Useable Database for Tactile Map Designers. Cartographic Journal, 40(3), 273-276. Ungar, S., Blades, M., & Spencer, C. (1993). The role of tactile maps in mobility training. British Journal of Visual Impairment, 11, 59-62. Uttal, D. H. (2000). Seeing the big picture: map use and the development of spatial cognition. Developmental Science, 3(3), 247-264.