Participatory Mapping in Congo-Brazzaville (Part 2): Usability experiments

Towards the last part of our field trip in Republic of the Congo and after eliminating most of the platform bugs and the decision tree became stable, we introduced the users to more structured usability tests to evaluate the effectiveness of the bespoke mobile application. The tests were composed of multiple tasks of different level and difficulty in order to quantify the efficiency (how much time users spent on a task and in how many steps they accomplish the given goal), the accuracy (the percentage of correct tasks in relation to the total tasks), the recall (the amount of information that the users do remember after some time period of absence of use) and finally the emotional response (how do the users feel after the completion of each task) of the participating users.

Some of the tasks included simple goals such as recording the location of the village and the location of important local resources (palm trees, banana trees, etc). As the tasks progressed in difficulty, participants were asked to augment the location data with photos or audio recordings depending on the question. The questions were formulated in such a way as to test users’ ability to judge in which situations it was appropriate to take a photo, record audio, or do neither. For the final task participants were asked to use the app to its full extent (including appropriate use of audio/photo), for instance to document a hypothetical situation in which the logging company had not delivered on a previously agreed upon compensation.

Soon we realised that conducting usability tests outside of a controlled environment, imposes a plethora of challenges:

• Time constraints:

  • Usually we were only able to spend about 3 hours in each place, during which we needed to conduct both the introduction, training (see part 1) and the actual usability tests. This is a very short time, especially considering this was often the first time participants used a mobile phone.

  • Participant usually also had limited time themselves, due to other obligations (e.g. tending to their fields).

• Communication difficulties:

  • Most of the research team did not speak or understand the local language.

  • Often we had to go through multiple steps of translation (e.g. English → French → Lingala → Mbenjele), with the potential for meaning to be lost or added.

• Cultural difficulty:

  • A lack of a cultural understanding with regard to being evaluated or tested one by one. Consequently bystanders and even translators would often help participants when they struggled to understand or perform the tasks.

  • The abstractness of some tasks (e.g. documenting hypothetical situations).

• The many distractions and interruptions caused by conducting the experiments in the middle of a village.

The final outcome of the usability tests were less encouraging than we expected – in the sense that participants often performed poorly in terms of efficiency as well as accuracy. However, these results can be partially explained by participants having received only a little amount of training, difficulties of communication, and the pressure they were under due to being evaluated – some of them were performing noticeably better before the actual tests. The latter can also be verified by the fact that in most of the cases users that had more training or the opportunity to spend more time with the software beforehand, performed prominently better in comparison to others and clearly shows that with the appropriate introduction and training the software solution can be applied in and used by almost every community.