Usability of geospatial metadata or space-time matters

Little is known about how end users actually employ metadata to evaluate relevance of geospatial information objects (e.g., books, articles, government documents, maps, digital files, databases, and Web sites) and how this might differ from meeting non-geospatial information needs. While other studies have considered users in related contexts, we could not find any that applied well-established usability techniques to geospatial metadata usage for resolving geographic information needs. Here we discuss a recent usability study where we explored how users determine the relevance or potential value of geospatial information objects from metadata.

Background

Metadata - often defined as "data about data" -- usually contain information describing the object's intellectual content (e.g., title and abstract) as well as its externalities (e.g., cost and availability). We investigated three well-developed U.S. metadata format standards for spatial data: Federal Geographic Data Committee (FGDC), Government Information Locator Service (GILS) and U.S. Machine Readable Cataloging (MARC). MARC is the standard used by libraries; GILS records are required to be produced for a range of federal agency information products; and FGDC records are written in the specifically geospatial standard required of federal agencies. (From an agency perspective, GILS and FGDC are not mutually exclusive; both may be used to construct different records in differing contexts for the same object.) This study also employed Internet search engine results as additional forms of metadata for analysis.

Relevance is an elusive term, much like quality. It can refer to utility, pertinence, value and other such attributes. Following the work of Carol Barry in "User-Defined Relevance Criteria" (Journal of the American Society for Information Science, v. 45, no. 3, p. 149 (1994)), we operationalized relevance as the "willingness to pursue" an information object. Such pursuit may be exhibited by clicking on a Web link, for example, or visiting a library to examine materials. Similarly, users may indicate nonrelevance by abandoning an item's metadata or performing a new search.

Usability analysis of an information system or product assesses the degree people are able to use the system or product, as well as its usefulness in successfully completing tasks. Analysts evaluate process efficiency and product effectiveness. With roots in engineering and psychology, the traditional usability approach monitors the performance of a small sample of task workers, typically measuring performance times and error rates. More recent methods broaden the limited, mechanical assumptions of the traditional approach with deeper views of being human. Thus, these techniques also involve collecting and analyzing users' verbal expressions made while completing usability tasks and in focus groups.

The overall goal of the usability approach is to identify and suggest preferable alternatives to the weaknesses of an information system, product or interface as noted by users. Issues noted should include both usableness and usefulness of concepts and functions. This supports development and deployment processes by gauging the ease of learning and use, and the degree of user satisfaction.

An Approach to Metadata Usability

Since many of the underlying issues of geospatial metadata usability are not known, we conducted a two-phase exploratory usability study. The first phase involved 10 academic participants (geography and information studies [GIS] graduate students and faculty), while the second phase involved 10 GIS or library professionals. As a group they brought a range of expertise from their respective fields.

In both phases each participant performed a series of four searches of metadata in a usability center laboratory. Three search topics were directed by the researchers using a limited set of previously collected and formatted FGDC, MARC and GILS metadata search results. The directed searches involved participants reviewing, assessing and reacting to potentially relevant metadata records using a Web browser to display them on a computer screen. These search topics sought information on hurricane surge heights in Florida, foreign travel safety and information on the relationship of hazards to growth management in the southeastern United States. The fourth topic was an individual search of the participant's own choosing executed in real time on the Internet. Participants used an Internet search engine accessed through the same browser. All participants were comfortable with the basic search tools. Subsequently, each group of 10 discussed geospatial metadata issues in separate focus groups.

As participants conducted their searches, they and the computer screen were audio- and videotaped. One researcher assisted the participants, encouraging them to talk aloud, asking them for clarification of their actions and probing for their reactions. The probing focused upon whether and why the participant believed a particular metadata record pointed to a relevant item. Another researcher watched and listened behind a one-way mirror, taking notes on usability-related aspects of the participants' metadata reviews. The focus groups were also audio- and videotaped. The data analyzed included the videotapes and selected transcripts of search sessions, observation notes, focus group transcripts, and participants' background and debriefing forms.

Observations of Metadata Preferences

The MARC, GILS and FGDC metadata standards all focus on specific content items to include in a records representation of a given document or data set. The impact of those content choices is far reaching and has justifiably received considerable attention. Comparatively little attention has been focused on what we refer to as metalevel usability issues regarding metadata, such as interface layout, presentation features and related aspects of display formatting. In usability analysis, the perspective of users is paramount, and every participant in our study indicated a concern with metalevel issues greatly disproportionate to the attention given them by others in the field to date. Our findings suggest that the metadata standards bodies have an opportunity to further their standards' use and usability in geospatial information retrieval by focusing attention on these concerns.

Metalevel issues of the metadata make the first impression of the underlying data on users. In our study, some impressions were favorable; others were not. This is significant not as an end in itself, but because presentation of the metadata has a direct and lasting bearing on whether and how the metadata content is evaluated. For instance, one participant characterized the overall "tone" of GILS records to be helpful, especially as compared to both FGDC and MARC records. Thus, the presentation alone may make the metadata more or less usable. For different reasons and at different stages, most participants encountered metalevel irritants which tested the limits of their tolerance: beyond a point, they were unwilling to sort through the perceived noise for what might have been highly relevant data.

Format: The importance of readability issues and related aspects of format 'became apparent early in the study as our participants expressed preferences regarding boldfacing, indentation, outlining, line spacing, capitalization, fonts and font sizes. The collection of metadata records made available for participant viewing contained a variety of format styles, and most participants readily volunteered their views. Some of the samples included, in various combinations, headings and/or subheadings that were boldfaced, tabbed or indented; and text that was tabbed or indented, single- or double-spaced, or all in capital letters (see, e.g., figures 1 and 2).

The focus groups gave participants an opportunity to compare the three different standards and presentation styles for essentially the same data. Many of the comments revolved around format, style and general impressions. A participant in one focus group stated that the boldfacing of headings seen in the MARC record would have made the GILS record "much nicer and easier to follow." This was echoed in the other focus group in reference to the FGDC record: "[Boldfaced headings] would help me organize what I was looking at because it's a little bit confusing when you first pick it up.... If it was bolded and tabbed and better organized in terms of the structure of the document, then it would make it easier to read."

Another participant offered that: "As far as readability goes I think the GILS is the clearest and perhaps the most pleasant, if I can use that word, to read. It's very clear." This participant had a very different reaction to the FGDC record: "It's pretty ugly [because it has] no white space [and] a lot of more technical terms that I am not prepared to deal with though for the professional it might be better. I can find information in here. I can find where to get my material from...but it's a struggle."

In comparison to the "narrative" characterization of the GILS record, one person noted that the FGDC "outline form in some ways is not as friendly, especially if you don't have the proper tabs and spaces.... He then notes, however, that when consulting a specific field of a record in outline form, the user can be helped by being "reasonably certain that [information] really isn't there or that it is."

Stylistic preferences can be quite personal, and this study did not attempt to catalog the range of participant reactions to various formatting alternatives. However, one issue evoked some unusually strong responses during the searches: full capitalization (see, e.g., figure 1). One participant commented on the perception of such text and the effort required to cope with it: "[Text] printed all in caps comes at me all as one entity. I would have to break it up for meaning. If I really needed this answer I might spend more time on it and look to see...." In other words, she would rather avoid making the effort to decipher such text. She later added: "I find it irritating." Another participant exclaimed: "It's not appealing. [This text]...all in caps...[is] boring me to death." Many of the participants' contemporaneous gesticulations and facial expressions were no less negative.

It would be difficult if not impossible to identify a universally pleasing presentation style. Nonetheless, it is clear that metadata designers must pay more attention to typographical and other readability issues if they want users paying more attention to their content.

Length: In the same way that display formatting can help or hurt a user attempting to read textual content, the sheer length of the record and individual fields can have similar impacts. Of course, the length of an entire record or content units is dictated in part by the relevant metadata standard and the data being represented. However, length of metadata material can produce a visceral response in users analogous to presentation style reactions.

At one end of the spectrum, the quantity of information provided is so meager as to leave the searcher essentially in doubt of the information's value. For instance, one participant remarked: "The problem is I don't think a lot of those [MARC] bibliographic pieces have enough information about their topics. They're almost too general." While this criticism is one that may be viewed as being directed at the MARC standard, the standard could accommodate the provision of greater information, with more usability and utility in the eyes of that participant; however, for a variety of reasons, including the custom and practice of Library of Congress subject heading construction, it does not. This results in descriptions so brief in some cases as to be cryptic.

Quite often, the small quantity of metadata gave users the impression that the data was not worthy of further consideration, since the metadata was not amenable to analysis. If their impressions were counterproductive in that the underlying data could have helped them answer their queries, the impressions were nevertheless derived from metadata that insufficiently represented that data. Brevity left these users confused as to the potential usefulness of the data.

The other end of the spectrum is problematic as well. Large volumes of metadata can be quite valuable to some users sometimes, but many participants were overwhelmed by the size of the FGDC record reviewed in the focus groups (see figure 2). One of them stated: "I don't think that anybody... unfamiliar with the federal standards is going to have the patience to go through twelve pages of this.... I would just feel I would get tired like in the middle of this sixth page looking for the stuff that I really, really need." Another conveyed a similar feeling: "I mean when you're giving somebody 12 pages, you might as well give them the document."

The length of individual attributes in a record presents the same issue as with the record as a whole. This particularly applies to keywords, descriptors, subject headings and abstracts: if too brief, they convey the instant message that little help has been provided; if too long, they immediately frustrate the user with data overload.

As with our observation regarding format, there is no magic length for all records or fields that always satisfies all users. The only clear lesson is to avoid extremes.

Order of Attributes: Due to the high value all participants placed on efficiency when making relevance evaluations, titles and abstracts (when available) were invariably key metadata attributes and were considered immediately if possible. Without exception, titles were located at or very near the top of every record, as is typical. However, the location of abstracts varied substantially, particularly in the GILS records. Tentative relevance judgments were often made very quickly by participants first viewing a record with an abstract at or very near the top, while many expressed frustration with using records that had abstracts located below the initial screen, particularly if the record was long. Most participants expressed a clear preference for ordering attributes with abstracts very near the beginning and a similar preference for keywords, descriptors and subject headings, after abstracts.

Once a user has decided to pursue the underlying data, the metadata would ideally facilitate access to the data by allowing quick identification of logistical information (e.g., location, contacts and price, as appropriate). Our participants demonstrated some frustration and confusion with the varied placement of these attributes in GILS records, and there was some frustration with finding that information in FGDC records, despite the standard structure of such records. Thus, a number of participants expressed a preference for an order of access-related attributes (e.g., item availability) that approximates the sequence when users are likely to want that information.

Geographic Resolution: All searches had a spatial component. However, many participants found that matching queries to spatial attributes was problematic. In some cases searchers identified insufficient information in the metadata to determine whether the location embedded in a query was contained in the object being represented. This uncertainty made it difficult for them to decide whether they were willing to pursue that object. For example, the researcher-directed query regarding the environment asked, "How has the presence of radioactivity or toxic waste affected growth management efforts in the southeastern United States?" Of the various titles in the prepared search results list, most participants initially commented favorably on one particular title, Growth Management: Issues, Techniques, and Policy Implications. However, once the MARC record for that title was viewed, the lack of information to identify whether the referenced document covered the area in question made relevance determinations difficult. As one participant expressed in frustration: "Well, it doesn't say whether it's southeast or not."

Time: All participants expressed a natural desire to achieve success in their searches with the least effort possible, and in the various searches time played a significant role in at least two basic ways. First, the dimension of time was elemental in that users had to locate both current and historical data. Failure of the metadata to provide adequate time-based cues for users to discriminate among search results was totally unsatisfactory to our participants. Materials that users judged potentially interesting based on their titles and/or other metadata were not pursued when dates of publication or other time-related attributes were unacceptable or uncertain.

Time also revealed itself in a second, more fundamental way: as a limited resource. Through words and actions participants demonstrated a perception and consideration of available time, and of demands on their time, that profoundly influenced their general strategies and individual estimations of relevancy. Each explicitly or implicitly approached the tasks at hand with a keen sense of time usage.

The key for all participants was the value placed on ease-- of-use. The significance of metadata's usability in supporting geospatial information searches cannot be understated: each individual had a strong desire to identify appropriate, meaningful data easily. However, to the extent the metadata was perceived nonusable in distinguishing potentially relevant from nonrelevant data, frustration tended to turn them away from considering the metadata, no matter how useful the underlying data might actually have been. When participants verbalized a sense that their time was being poorly used when evaluating certain pieces of metadata, their estimation of the value of the metadata, and thus the data, sank dramatically.

Even if a search is ultimately successful, metadata, which wastes a searcher's time, will encourage compensating strategies that can lead to pursuit of greater efficiency at the expense of effectiveness. Thus, for instance, long records with key fields (such as abstracts) buried among other fields not clearly structured might result in quick exclusion by the user; thereafter, records similar in general appearance might be excluded automatically in favor of records more readily scanned, although more likely lacking in desired information. This sense of increased search speed can actually increase time-wasting review of thin records or pursuit of data not properly excluded at the metadata stage. The success that metadata will ideally support is truly efficient and effective fulfillment of information needs.

Metadata Design, Metalevel Issues and Time

Given that metadata designers cannot anticipate all information needs to which data and corresponding metadata might be applied, it is important to make metadata consonant with users' expectations of ease-of-use and hopes for easy success. From the perspective of our participants, some issues transcend the particulars of one format and apply broadly. Our study suggests the following are fundamental to the usability of any geospatial metadata format and should be given consideration in metadata design:

Titles and abstracts are extraordinarily important to users when evaluating whether to pursue geospatial data;

Readability, in all its variety and subtlety, matters greatly;

Length or quantity of information is important, as users want just enough, but not too much, to make the quickest possible relevance judgment;

Scalability of detail is desirable, allowing users to quickly access greater or lesser degrees of detail as desired, including details of geographic resolution;

Order of metadata attributes should match the order in which a user would want additional information, because users' domain issues (willingness to pursue for content) will normally precede their logistical issues (such as price and access points);

Allowance for flexibility in search styles is desirable;

Serendipity is frequently a major factor in search success, and metadata ideally would promote this possibility; and

Time is precious.

This article is adapted from the author's panel session presentation at the 1998 ASIS Annual Meeting and was funded in part by a grant from the LISRG program of OCLC, Dublin, Ohio.

Bruce Fraser is a doctoral candidate at the School of Information Studies, Florida State University, Tallahassee, FL 32306-2100; [email protected]

Myke Gluck is director of the Florida State University Usability Center and an associate professor at the university's School of Information Studies 244 Shores Building, Florida State University, Tallahassee, FL 23206-2100;850/644-8118; [email protected]

Copyright American Society for Information Science Aug/Sep 1999
Provided by ProQuest Information and Learning Company. All rights Reserved