The mobile Vulnerability Analysis and Mapping (mVAM) project was designed to both collect data from World Food Programme (WFP) beneficiaries and share critical information with them. Since its inception in 2013, mVAM has developed SMS, interactive voice response (IVR), and computer assisted telephone interviewing (CATI) techniques to allow WFP country teams to provide beneficiaries with access to free information, collect feedback, and monitor food security and nutrition trends. In 2016, mVAM engaged nonprofit technical partner InSTEDD to begin experimenting with chatbots as an additional channel for exchanging information with beneficiaries. The mVAM chatbot concept would undergo multiple iterations but eventually become known as the Food Bot, the underlying structure of which InSTEDD planned to create as an open-source tool that would benefit the entire humanitarian communit.
While early experimentation in 2016 began with a prototype built on Telegram, mVAM and InSTEDD quickly shifted to working with Facebook Messenger after a site visit in Kenya found few beneficiaries were familiar with Telegram. In March 2017, mVAM and InSTEDD both sought and received funding from the Cisco Foundation to continue developing the mVAM Food Bot concept in three countries as the basis for a more generic open-source humanitarian chatbot tool. Yet by mid-2017, the two partner organizations had concluded that a single generic Food Bot would be insufficient to meet WFP’s diverse needs and decentralized structures. Users in different countries—both beneficiaries and country teams—accessed messaging applications in unique ways and had different information and communication needs.
What mVAM really needed, and what InSTEDD viewed as even more valuable across the humanitarian field, was a web platform on which humanitarian teams could quickly and easily build and deploy their own custom chatbots with little in-house programming or IT skills. Together, mVAM and InSTEDD committed to developing, deploying and ultimately opening a chatbot builder platform designed specifically for the humanitarian sector called AIDA. As the co-designer of AIDA, mVAM would become its first major user, enabling each of its country offices to develop their own chatbot tools in local languages and to align with their local food security needs and initiatives. In late 2017, InSTEDD began developing AIDA, with plans to refine it after mVAM had deployed across multipIe countries in 2018, then release the platform as an open-source tool at the end of the year.
Key Lessons |
|
|---|---|
|
During mVAM and InSTEDD’s preliminary user testing in Kenya, Nigeria and Haiti, a number of key lessons emerged regarding the deployment of chatbots for development: |
|
|
1 |
Simplify questions and replace jargon with slang and colloquial language to make users feel they are talking to an actual person and not a robot. |
|
2 |
Make the user experience more intuitive for people with low digital literacy. |
|
3 |
Ensure that the chatbot understands user misspellings. |
|
4 |
Develop a chatbot that consumes as little data as possible and works on Facebook Messenger Lite and on all internet-enabled phones. |
|
5 |
Consider that chatbots may be most useful for the transmission of non-sensitive data. |
|
6 |
Follow the “security by design” principle to ensure data security and privacy (e.g., through encryption and account permission settings). |
|
7 |
If a project works across various countries, ensure that each country is able to collect data through a chatbot that is useful in their own local context. |
These include Comprehensive Food Security and Vulnerability Analysis (CFSVA) and baseline food security and household vulnerability surveys. In rapid and slow-onset emergencies such as hurricanes, floods, droughts and conflict situations, VAM also conducts Emergency Food Security Assessments (EFSA).
In 2013, the VAM unit created the mVAM team, designed to provide VAM with alternative tools to collect data remotely. Using short SMS surveys, live phone interviews, online surveys and IVR systems, mVAM tracks food security and vulnerability among beneficiaries. The resulting data is then cleaned, anonymized and analyzed in order to make food security trend data and analysis available for WFP’s programmatic decision making as well as to the public. mVAM also seeks to make critical information available to its beneficiaries free of charge. Using IVR and free websites (Facebook Free Basics platform where available), beneficiaries can access information about food prices and WFP services such as food distribution dates and location, as well as provide feedback to WFP, all for free.
In 2016, the mVAM team decided to experiment with incorporating messaging applications and chatbots as yet another channel for beneficiary engagement and information exchange. mVAM conceived of the Food Bot as an addition to the team’s toolbox of mobile communication tools for food security analysis. Across WFP’s beneficiary countries, mVAM had observed the growing popularity and accessibility of messaging applications, particularly among young people. Engagement with them via a chatbot would be cheaper for WFP than SMS or phone calls, and while mVAM had historically developed one SMS or voice system for collecting information and another for beneficiaries to access information, chatbots would enable both through a single channel and user interface.
Moreover, a messaging chatbot would allow WFP and its beneficiaries to exchange far richer and more dynamic information than other channels, and mVAM expected that a chatbot would greatly increase WFP’s speed and responsiveness. Beneficiaries would be able to send long complex sentences, photos, voice notes and geolocation tags, all of which were impossible with phone calls, SMS and IVR. mVAM also saw an opportunity to better integrate real-time data, so that users could query WFP’s food price databases and immediately receive the most updated information. This would be a significant improvement on the IVR and SMS systems, which previously responded with static pricing data updated weekly at most. Lastly, mVAM expected that messaging applications could eventually overtake SMS and voice as the most commonly used mobile communication channels among WFP beneficiaries. Experimentation with chatbots and messaging applications was, therefore, seen as a timely investment in the future.
To pursue its vision for a Food Bot, in early 2016 mVAM engaged InSTEDD, a technology nonprofit and founding partner of mVAM, which had repeatedly deployed mVAM’s IVR and SMS platforms across multiple countries. InSTEDD’s mission is to design and develop new, unique, open-source technology to meet social-sector needs. Having already begun internally experimenting with chatbot technology, InSTEDD saw the mVAM partnership as an opportunity to develop the initial architecture for a more broadly applicable tool—a chatbot for humanitarian response.
Prototype development for the Food Bot began in June 2016 by trying to replicate the same services mVAM offered through SMS and IVR. This meant that beneficiaries with Telegram would receive periodic one-to-one messages from the Food Bot, consisting of an invitation to take a survey. If they responded affirmatively, they would receive a survey with several questions about their food security and livelihoods. Users would also be able to use the Food Bot to query a database and automatically find local food price data at any time. Initial testing of these two use cases was conducted with WFP staff in Rome and InSTEDD staff in the United States.
In August 2016, mVAM gathered feedback from users more closely resembling actual WFP beneficiaries, turning locally to Rome’s refugee population, which had expanded significantly following conflicts in Africa and the Middle East. The mVAM team held a focus group discussion at a migrant center in Rome to collect user experience feedback on the Food Bot prototype and how it might be useful in the refugees’ home countries. Participants were first asked about smartphone ownership and use in their home countries. They reported that smartphone ownership and usage is very common in their communities. The participants were then asked to test the Telegram Food Bot prototype on their phones, first answering a survey and then trying the food price database. They provided a range of feedback, with recommendations to:
Simplify questions and replace WFP jargon with slang and colloquial language to make users feel they are talking to an actual person rather than a robot.
Make the user experience more intuitive for people with low digital literacy.
Ensure that the chatbot understood user misspellings, which were very common.
Make the food price information specific to the user’s local context so that users could use it to locate the lowest prices.
Use a different messaging application, because few people in their countries use Telegram.
In January 2017, to generate new insights and develop some of the necessary upgrades identified by the focus group participants, mVAM turned to Nielsen, a global information and measurement company that has long provided WFP with skilled volunteers and in-kind technology and development input. To help upgrade the mVAM Telegram chatbot, Nielsen organized a 24-hour public hackathon in New York, which attracted developers, students, volunteer hackers and Nielsen staff. The hackathon participants first brainstormed a series of critical upgrades in direct response to the feedback from refugees, then divided into teams to build out the new components in real time.
The new components included multiple gateways to different messaging applications, including Facebook Messenger; natural language processing capabilities to manage misspellings; and reporting and data visualization features so that mVAM analysts could view survey results. As part of the event, the mVAM team and participants also engaged with Alex Lazarescu of Chatbots Magazine, who recommended focusing on content, such as onboarding messages to clarify what the Food Bot was for and options for users to chat with a human if they became stuck.
In early 2017, the mVAM team visited Haiti, Kenya and Nigeria to test the new Food Bot in the field. mVAM assembled focus groups and conducted in-depth interviews with community leaders, women and youth, finding quickly that even in hard-to-reach communities, community leaders and young people owned smartphones and were connected to the internet. Across all three countries, it was clear that Facebook Messenger and WhatsApp were the most popular messaging tools, especially for young people and community leaders. With this in mind, mVAM had participants test the new Facebook Messenger version of the Food Bot (facilitated through Chatfuel).
While the majority of users described the Facebook version as a convenient, quick and easy way to get in touch with WFP directly, it did not function well on Facebook Messenger Lite or on the web browser version of Facebook Messenger. This was because Chatfuel relied on Facebook Messenger features that were specific to the Facebook Messenger mobile application and, therefore, not available to those accessing the service through other means. This was problematic for the many beneficiary users who relied on Facebook Messenger Lite and logins via their mobile web browser in order save money on data.
In March 2017, mVAM and InSTEDD received funding from the Cisco Foundation to continue developing the mVAM Food Bot concept in the three test countries. The intent remained for the Food Bot to form the basis for a more
generic open-source humanitarian chatbot tool. After making upgrades during the summer, the mVAM team returned to Kenya in September to conduct further research and testing. These second visits were used to engage
with WFP’s country teams about how a Food Bot could best provide value and help meet local needs.
The team also focused on content development, working with small groups of refugees to better understand how they would interact with the tool, and what type of information they might seek or provide. To
do so, mVAM and InSTEDD used a rapid prototyping technique to effectively simulate the experience of interacting with a chatbot. Group participants were asked to visit a Facebook page and start a conversation with
the Food Bot, but what appeared to be a bot was actually WFP and InSTEDD staff members manually responding to the messages with predetermined responses. These conversation transcripts were then used to help optimize
the Food Bot’s content, response handling, conversation flows and tone.
mVAM’s continuous and iterative user engagement, testing and research in the field during its initial Food Bot development revealed a wide range of complex challenges related to the diversity of contexts in which WFP works. These challenges ultimately highlighted that a single, globally available mVAM chatbot was not the correct solution and instead a flexible chatbot builder would be best.
|
Phone access: |
In the 25-year-old Kakuma Refugee Camp in Kenya, where many residents were born and raised, 90 percent of households had access to either a basic mobile phone or smartphone, and internet was available in some locations. Phones were considered by most residents as critical for communicating with friends and family in their home country. Conversely, in the newer nearby refugee settlement of Kalobeyei, which houses new arrivals, less than 20 percent of households had a phone. Charging was difficult and expensive in Kalobeyei, and while many young people claimed to have Facebook accounts, few could access them as they had not yet obtained a Kenyan SIM card and struggled to do so without the proper identification. |
|
Connectivity and affordability: |
In all contexts, the Food Bot was found to be responsive on 3G networks and even on some slower 2G connections. However, in Kenya and Nigeria, most refugees and internally displaced persons (IDPs) struggled to pay for mobile internet data due to their limited earning capacity, which made it more difficult to engage in lengthy or consistent conversations via Facebook Messenger. In Haiti, where there is a lot of competition between mobile operators, the cost of mobile data bundles was relatively lower, making it possible for most of the community leaders to consistently access internet on their smartphones. |
|
Messenger access: |
As a result of the high cost of mobile data, mVAM found that Nigerian IDPs and Kenyan refugees accessed Facebook Messenger in different ways. While some used the regular Facebook application, others used the Facebook Messenger Lite mobile application or the Facebook website on their mobile web browser in order to use less data. This created problems for mVAM’s initial Food Bot design, as different access points for Facebook Messenger support different sets of features. For example, for those using their mobile web browser and Facebook Messenger Lite, multiple choice buttons did not appear within the chat dialog box. For a single, generic WFP chatbot to be viable, mVAM would have to use the bare minimum of user interface features. |
|
Language: |
In Haiti, the Food Bot was made available in French and Creole. In Maiduguri, Nigeria, it was first demoed in English, but the young Nigerian users overwhelmingly requested it be translated to Hausa, the most common local language. In Kakuma, the mVAM team anticipated that most people would require translations in Somali and Dinka, the most common languages among Somali and South Sudanese refugees. However, interviews with young refugees revealed that most of them had been born in the camp and preferred English or Swahili. These language preferences highlighted the need for the Food Bot to communicate in different languages. |
|
Utility for users: |
The focus groups revealed that young displaced people in Nigeria were most interested in having the chatbot be a new, more efficient way to access information about WFP programs and services as opposed to posters, hotlines, help desks and loudspeakers. In Kenya, on the other hand, young refugees were most interested in a chatbot as a direct line to WFP for providing feedback and complaints. Outside the refugee and IDP context, users in Haiti struggled to get value from the Food Bot, interacting as if it were a human and getting stuck quickly when the chatbot could not understand their conversational inputs. This highlighted to mVAM the importance of clearly conveying from the beginning what a chatbot can and cannot do, possibly through offline sensitization campaigns. In all of the focus groups, the Food Bot’s utility was seen as limited to nonsensitive topics, as users and WFP staff agreed it could not be trusted as the right communication channel to ask sensitive questions In addition to this diverse array of needs and challenges among beneficiary users, mVAM discovered that each WFP country office also had vastly different needs and expectations for a chatbot, which reflected their different programmatic goals and contexts. In Kenya, the country office wanted a new way for beneficiaries to submit feedback and complaints to WFP about its services. In Nigeria, the team felt a chatbot would be most useful for collecting price information from traders. |
In September 2017, following its user research sessions, mVAM and InSTEDD discussed their findings to determine the way forward. The teams mapped all possible functions and applications for a Food Bot, attempting to identify priorities for each country office. They ultimately decided to switch directions. Rather than develop a single chatbot on a commercial platform designed to serve both internal users (country offices) and external users (beneficiaries), they decided to build AIDA, a web platform to allow WFP’s different country teams to develop their own chatbots, customized to their local context. Just as with the original Food Bot concept, mVAM would serve as AIDA’s co-designer and first core user. The team would help WFP deploy the platform across multipIe countries in 2018. InSTEDD would then refine and ultimately release AIDA as an open source tool at the end of the year, in keeping with the funding mandate from Cisco Foundation and InSTEDD’s mission.
Based on user testing, mVAM and InSTEDD were convinced that to be effective, AIDA should not require any in-house programming by WFP’s country teams, which like most humanitarian country teams rarely had such capacity. This immediately eliminated most commercial chatbot platforms such as Chatfuel, which would have required too much coding for WFP country offices to be able to customize their own chatbots. mVAM also concluded that a commercial platform could not provide the data security required to protect vulnerable beneficiaries, nor were they optimized to provide consistent user experiences across Facebook Messenger Lite, the Facebook Messenger application and the Facebook platform accessed via a web browser.
Thus, with the original Cisco Foundation funding and additional support from the Korea International Cooperation Agency (KOICA), mVAM and InSTEDD set to work in October 2017 to design AIDA, with a first round of development ending in February 2018. As with the original Food Bot concept, AIDA is still intended to enable data collection as well as provide information about WFP programs, food prices, weather, nutrition and disease prevention. AIDA’s first iteration will integrate with Facebook Messenger, with the idea of later integrating gateways to other messaging applications.
Each country office team, and eventually any humanitarian team from any organization, will be able to use AIDA to create a local Facebook page on which they directly activate their Facebook Messenger chatbot. On AIDA, teams will then be able to create custom conversation flows, keyword, surveys and scheduled messaging, all in whatever local languages they choose.
|
Why user testing led mVAM to develop a chatbot builder platform |
|
|---|---|
|
Beneficiary Users
|
WFP Users
|
|
Pivot: Build a new web platform where country teams can quickly deploy their own custom chatbots. |
|
As of March 2018, InSTEDD and WFP had developed a working AIDA prototype for testing in Kenya and Nigeria. Following this preliminary testing, additional features will be added and testing expanded to new countries and contexts. During this second stage of testing, mVAM will test the platform’s flexibility to allow country teams in more drastically different contexts to rapidly create and deploy a chatbot.
By late 2018, InSTEDD expects that AIDA will be flexible enough to support the needs of different WFP offices, but mVAM still anticipates an extended period of hands-on support. With this in mind, in 2018 every WFP country office that wishes to use the early version of the platform will be asked to identify one focal point to receive training and sustained support from the mVAM team in Rome. mVAM will produce public user manuals, instructional YouTube videos and live webinars on the use of AIDA, all of which will be made public and open-source alongside the AIDA code base by the end of the year.
During 2018, mVAM planned to evaluate the Facebook Messenger chatbots built on AIDA by measuring engagement, retention and churn rates across different country teams, as well as the number of contacts and the level of satisfaction among users in different countries. In the longer term, InSTEDD planned to expand AIDA to integrate with a variety of messaging applications. Both mVAM and InSTEDD are hopeful that this will include WhatsApp. As of March 2018, WhatsApp did not have a publicly available API to enable integration with other platforms like AIDA, but was actively piloting an enterprise solution with select organizations, including MomConnect, a public health platform in South Africa.
In addition to multiple gateways, mVAM and InSTEDD hope eventually to enable AIDA to deliver more than just automated, structured conversations and incorporate technology using natural language processing. While in 2017 InSTEDD had begun testing wit.ai, a natural language processing product that can support Swahili, many of the other local languages spoken by WFP beneficiaries were not supported by existing artificial intelligence products.