In recent months, my company Stamplay worked together with IBM to create a chatbot for a small town in Tuscany, Poggio a Caiano. The initiative was promoted by the vice-mayor Francesco Pugelli, IBM provided its natural language processing and conversation management technology, while the Stamplay team used our platform to develop the application logic and the integration with Facebook Messenger. During the project we learned some things that are worth sharing.
Managing expectations
When we talk about artificial intelligence, people’s imagination immediately goes to science fiction movies. The recurring expectation is that a chatbot is a software that can receive any documentation, process it magically or even understand it, and answer any questions concerning the documents it has processed. Alas, it does not work like that. The technology is much simpler. Machines are able to classify texts in different categories but do not understand the content. Furthermore, training an algorithm requires many hours of manual work by a human. Taking these constraints into account, we decided to create the chatbot making three basic choices:
1. Answer only the frequently asked questions
We immediately abandoned the idea of managing complete conversations (because there is no system able to do it effectively) and we focused on question / answer exchanges. The system receives a question from the user, classifies it and then identifies the corresponding answer. More complex scenarios don’t produce a satisfying experience for the user, so they are not feasible today.
2. Address a municipal service at a time
In a Q&A system, any question that has an answer represents a category. The goals is to analyze any question that is submitted and classify it in the right category. If we are able to do just that, then we know which is the answer. Let’s assume that I know how to answer the question “where can I look at the menus of the school canteen?” A user can express this question in many different ways, and I need to recognize all the possible variations, to provide the same answer.
Training a classifier means supplying the system with a certain number of variants of the same question. The advantage of using an artificial intelligence solution like IBM Watson is that, once the examples have been provided, the machine should be able to correctly classify all requests with a similar formulation.
Text classifiers have various limits. In the case that we have considered, the most important is that they can’t work on very large sets of questions. In other words, a natural language processing system is effective with a hundred different questions (and related variants), but as this number grows, the precision will inevitably tend to decrease. And therefore the usefulness.
3. Launch in more steps and learn
Since we didn’t know how users would behave, we decided to develop the chatbot in successive steps. We built a first set of about thirty questions about the public school of Poggio a Caiano, we published the chat bot and did a test with a small number of citizens with kids. We collected the questions, checked that they were correctly classified, and corrected the errors. It is a tedious job that is often underestimated. The reality is that the fine-tuning activity must be constant and implies choices, including deciding not to answer some variants of a question.
The next steps
We have been publicly online for a few days and we are collecting more data on interactions with users. The next step will be to add a navigation tree to start handling questions concerning multiple public offices. In this way, the user can choose the topic first and then ask a question in natural language.
Thanks to the partnership with IBM, we intend to explore the opportunities to automate the requests that are made to the public relations offices (URP) of the Public Administration, with the aim of helping the citizens to extricate themselves more easily in the meanders of Italian bureaucracy.
If you want to ask a few questions to Filippo, find on the Facebook page of the town of Poggio a Caiano.