Book Review: The Semiotic Engineering of Human-Computer Interaction
By Gerd Waloszek, SAP AG, User Experience – 12/16/2005
This review takes a personal look at Clarisse Sieckenius de Souza's book The Semiotic Engineering of Human-Computer Interaction.
Book
 |
Clarisse Sieckenius de Souza: General |
Author
Clarisse Sieckenius de Souza is Associative Professor and Founder of the Semiotic
Engineering Research Group, Department of Computer Science, at Pontifícia
Universidade Católica do Rio den Janeiro, Brazil. There, she started
the Human-Computer Interaction (HCI) area and gave origin to Semiotic Engineering – a
semiotically-based theory of HCI.
(From book cover and Website)
Homepage: www-di.inf.puc-rio.br/~clarisse/
Review
Introduction
De Souza starts her book on the semiotic engineering of human-computer interaction with the remark: "'Semiotic' or 'semiotics' are terms not frequently found in the HCI literature." This fact made me curious when I saw her book on the shelves at the CHI and INTERACT conferences. In the meantime, I've read the book and reviewed it – it was definitely one of the hardest reviews that I have had to write.
Overview
De Souza's book is divided into three sections. The "Foundation" section provides the background to semiotic engineering. Chapter 1, "Introduction," provides an overview of the book and of semiotic engineering, while Chapter 2 outlines – as its name says – the fundamental concepts in semiotics. It is "required reading" in order to understand the book and the theoretical concepts behind it. The third chapter is devoted to semiotic engineering itself and its relation to other HCI approaches, such as user-centered design (UCD).
In section two, "Derivation," de Souza provides three detailed examples of design problems with increasing complexity and demonstrates how semiotic engineering can be applied to practical HCI situations and which tools (called epistemic tools) semiotic engineering has in its repertoire. One method, for example, analyzes communicative breakdowns during interaction and tags these with standardized communicability utterances, such as "What's this?", "Oops!", or "Where am I?" All in all, I found the last example dealing with multi-user applications the most convincing.
Section three, "Questions," concludes the book with reflections rather than conclusions as a "testimony to my [the author's] ongoing semiosis," or – in simple words – the learning process. The author's reflections cover the relationship between semiotics and computation, between semiotics and design, and finally the question of epistemic support (knowledge support) for designers and users.
Digging into the Book...
Whenever a book introduces a new HCI direction, reviewing the book is difficult. On the one hand, it is the book that should be reviewed and not the approach, in this case it is semiotic engineering, as such. On the other hand, both are hard to keep separate and you are always in danger of mixing things up and discussing the approach instead of the book itself. Knowing this, I will walk, with eyes open, into this trap. First, I will introduce some basic concepts and the contributions to HCI of semiotic engineering, because I expect only a minority of HCI practitioners to be familiar with this new direction. In the conclusion of this review, I will add some remarks on the book itself.
Semiotic Engineering – What Is It?
De Souza's aim is to present semiotic engineering, a new direction in HCI which she gave origin to, as a viable alternative to, or better, as a complement to, current HCI approaches. In addition, she presents the tool box; she writes of "epistemic tools" that semiotic engineering has to offer to HCI practitioners. Probably, it is best to introduce de Souza's approach in her own words:
- Semiotic engineering is an attempt to bring together semiotics and HCI in a concise and consistent way, so as to support new knowledge organization and discovery, the establishment of useful research methods for analysis and synthesis, and also the derivation of theoretically sound tools for professional training and practice.
- The goals of semiotic engineering as a theory of HCI are to present an extensive and distinctive characterization of HCI, to provide a consistent ontology from which frameworks and models of particular aspects of HCI can be derived, and to spell out epistemological and methodological constraints and conditions applicable to the spectrum of research that the theory can be expected to support.
- A semiotic theory of HCI, such as the one proposed by semiotic engineering, has to incorporate signification, communication, and unlimited semiosis in its ontology.
The first two statements are fairly general statements with respect to the semiotic engineering research program and its goals. They basically tell us that contributions from semiotics, that is, the study of signs, play an important role in the approach. The third statement tells us more, even though this may not be obvious at first glance because the terms used are unfamiliar to the readers. Below, I will address each of the three terms used in this statement.
The New HCI Triangle
The "classical" HCI triangle consisted of the three components user, system (computer and software), and task. De Souza and semiotic engineering propose a new triangle consisting of the designer, the system, and the user. Thus, this approach gives a lot of emphasis to the designer (whoever that may be) and, particularly, to the HCI designer of a software application. Why does semiotic engineering give so much emphasis to the designer? Because semiotic engineering offers a new account of human-computer interaction, namely interaction is regarded as a communication process, or more precisely, as metacommunication. While the idea of human-computer communication is not new, the way in which semiotic engineering designs it is innovative.
The Designers Deputy
Probably the biggest problem with software that users have is that they often do not know how to handle it. One of the reasons for this is, and this would probably be the semiotic engineering answer, that users do not understand the designer's thoughts and intentions – and cannot find them out in (or through) the user interface. Thus, ideally the designer would sit next to each user and tell him or her what to do next and what his intentions were when designing the respective part of the application. This is, of course, impossible. Therefore, the designer needs a deputy – and that's just what the most important ingredient and contribution of semiotic engineering is: the designer's deputy, "played" by the computer system. Thus, "bubble two" in our triangle, the system, also assumes the role of the designer in the form of its deputy and tries to inform users about the designers intentions. This is the reason why I talked about metacommunication earlier, because the designer and user communicate only indirectly with each other. In de Souza's words, the designer's deputy is telling the user:
- "Here is my understanding of who you are, what I've learned you want or need to do, in which preferred ways and why. This is the system that I have designed for you, and this is the way you can or should use it in order to fulfill a range of purposes that fall within this vision."
De Souza continues: "This message is encoded in one or more signification systems (which introduces item one on the list above, signification – insert added by the author) specially designed to enable user-system communication and is conveyed through the system (channel or medium) itself."
I hear the alarm bells ring in the user-centered design (UCD) fraction. Are we in danger of reviving an idea that we believed to be long dead? Are we supporting the "users are stupid!" proponents, who put all blame on the users who are seemingly unable to decipher the designers noble plans? This is an open question for me. Of course, there are two sides of the coin, the other of which would be to attribute the users' failures to the designers because they were unable to communicate their intentions properly through the user interface. Nevertheless, my personal feeling is that the designer's role is overemphasized in the picture that semiotic engineering sketches of HCI.
Unlimited Semiosis
Next, I come to an interesting limitation of computer systems (and any formal system) that semiotic engineering points to, namely the limitations of computer systems with respect to their ability to express themselves, or better, the designers deputy's intentions (or discourse, as de Souza calls it). Says de Souza: "The designer's deputy can only reproduce a limited (rule-governed) segment of the designer's semiosis," that is, "the designer's deputy tells the user only a machine-processible version of what the designer really meant." Human beings, on the other hand, are capable of unlimited semiosis, which "says that the meaning of a representation cannot be defined as a fixed object or class of objects. The process is unlimited in the sense that no one can fully predict its duration, its path, its content, and so on." In other words, human beings, or in the case of HCI, users and designers, are capable of changing the meaning of objects forever, they are learning throughout their lives.
I do not want to go into the philosophical discussion of the meaning of meaning – de Souza's book is a good reference for this topic. Rather, I would like to point to an important aspect of user testing that is a direct consequence of this human ability: Users are a "moving target," they are never the same. For example, users may run into a problem during a usability test and this problem is marked in the resulting test report as a severe problem. But they may never run into this problem again because they were shown how to circumvent or solve it. Thus, many "severe problems" may only be "one-time" problems because users initially lacked certain knowledge. In the long run, other problems may show up because these users become more proficient and their focus is then on speed, not on "know how." I believe that current HCI (and practitioners) takes too little notice of the fact that users are learning all the time; the focus is too much on the problems of first-time users; these problems may vanish into thin air after the first few encounters with a software application.
To sum up, I believe it is important to clarify possible misunderstandings of the role of designers and users and to make clear – and get a common understanding of – what the valuable contribution of semiotic engineering to HCI is and where it falls short. Otherwise, we might as well be in danger of reviving old prejudices about users and their limitations. Let me conclude this section with an example of the latter: A theory, which clearly states that it has nothing to say about aspects, such as performance and efficiency, might have a hard time winning the hearts of HCI practitioners and software buyers whose main concerns are improving productivity, improving productivity, and improving productivity. Why should HCI professionals bother to learn new methods and terminology when the new approach does not supply answers to their questions?
Conclusion
It is not clear to me, which audience the book is targeted at. To me, the book's difficulty is that readers must be willing to learn and adopt the semiotic engineering terminology. This terminology is deeply rooted in linguistics and philosophy and is certainly not the native tongue of user experience practitioners. People who are already familiar with this terminology and theorizing or are willing to learn it may well profit from the book and gain many new insights. Those who are not are always in danger of giving up and laying the book aside – exhausted or even frustrated. Once you lose the thread, there is little chance that you will find it again. I even found the many recurrences of statements of little help because they were formulated in the language of semiotic engineering; this was, however, my personal encounter with the book and may not be valid for other readers. From my experience as a "user" of the book, I would like to suggest a way of improving its usability: Why not provide a glossary with the most relevant and often-used terms, either as a flip/fold out or a separate page? It is hard to scan a book for forgotten definitions that are distributed all over the book. You might stick Post-it®s on the relevant places, but that would not be very useful for me because I would need too many of them... Alternatively, you could open Wikipedia on the Internet and look for definitions but this would require a computer while reading. As a way out, I collected a "semiotic engineering glossary" for my own use and have provided it for the convenience of visitors to this Website (most of the definitions are taken from the book and from Wikipedia). Simply open the print version of the glossary and print it – or use the glossary online when you read the book if you do not mind having a computer at your side.
This finally leads me – after all this ranting – to the nasty question of whether I recommend the book and for whom. As this book presents a new direction in HCI – or one that is going to become one – namely semiotic engineering, this book is indeed relevant for everyone in the HCI field. But perhaps we should leave it at the moment to people who are interested in the methodology and philosophy of science-related questions, before all practitioners rush to buy their copies. Let me, as a consolation, add that the focus of this new approach is on online help and onscreen documentation, as well as on collaborative applications and applications, in which communication plays an important role in general. Thus, HCI people who are looking for new ideas and inspiration in these areas might also consider taking a closer look at this book.
References
