SP9: Design and Evaluation

The prototype "mobile device" that is to be functional for the 2006 FIFA World Cup is supposed to be evaluated in its effectiveness. Within the project SmartKom many extensive studies about the evaluability of multimodal dialogue systems have been carried out. SmartWeb can now draw on this expertise. SmartKom especially revealed that it is not sensible to let completely naive subjects evaluate the prototype. The results of these tests were not utilizable in a sensible way. In the same way it was proven that it is not sensible to test complex systems in the area of man-machine-interaction (MMI) in short-term experiments. Insofar there exists a certain dilemma in the evaluation of HCI-systems: on the one hand, there should be enough subjects available in order to reach statistically significant conclusions, on the other hand it turned out that results from long term studies where the subjects had time and opportunity to really test all the possibilities of the system, delivered much more valuable insights.

As soon as the demonstrators are available, the realized functionality (application/display, adaptivity) shall be tested according to their applicability and usage in a car. Accompanying, acceptance judgments will be raised.

According to the competences of the project partners, research will be conducted with subjects in the usability lab from BMW or on a certain proving ground. A special challenge will be the evaluation of user or situation adaptive features, as this question is not or only very insufficiently addressed by established evaluation methods. Therefore, an efficient testing alternative to long term tests shall be specified and tested.

New mobile and multimodal services will only be an economic success when they are adapted to the wants, expectations, and abilities of the users, meaning that it will only work if their development regarding functional range and design of the user interfaces and regarding service quality will be user oriented.

© Webmaster
Last modified: Thu Mar 3 11:30:09 CET