by Part of my job description is our CI/CD and it kind of implies that I’m interested in keeping the build green. It doesn’t mean that I immediately jump in whenever some unit test fails, but I’m definitely keeping an eye on unreliable ones.
Whenever master branch stays red long enough, this is what starts to happen to each failed test in it:
- Look for test failures history in Google BigQuery (
select Name, Result, count(*)...). - If test behaves like a random results generator, create a case for that.
- Skip the test in master branch and put the case number as a reason.
- Find out who created the test (
git blame) and assign it back to the author.
Pretty simple. And boring. I can automate that, but it’s not always clear who is the author of the test. After all, people resign, update each other’s tests, refactor and destroy git history on special occasions. I was thinking about doing something with machine learning to solve that, but it feels like an overkill. Creating a bot, on the other hand, who would ask me to double check when it’s uncertain, sounds more interesting and actually doable. Even if I’m never going to finish it.
However, I’ve never wrote any bots before, so for starters I’d like to check what it actually feels like.
Choosing a bot framework
Easy googling reveals that Microsoft Bot Framework seems to be default choice for bot writing. There’s also BotKit, but having Microsoft behind the tool might mean that there will be more examples for it. Plus, Bot Framework supports both C# and JavaScript – the two languages I’m the most comfortable with, so the choice was easy.
On the surface, developing with Bot Framework seems simple. For instance, in its object model there’s a Bot object, which has Conversations with Users through different Channels. Single Conversation consists of one or more Dialogs, and Channels are basically the tools User uses to interact with the bot: chats, Facebook, Skype, Slack, etc. Quite logical. There’re also some other object types and concepts behind it, but you get the idea.
However, despite such object model does look logical, writing with it is not quite intuitive. Few times I took existing bot example, made a tweak or two, and then dive into documentation trying to understand why exactly it doesn’t work. And I can tell you, the level of details I had to get into was incomparable to triviality of the change I made. But maybe it’s just me.
Writing “Hello World” bot
“Hello World” equivalent in the bot world is the bot echoing every message back to its sender. For this one I used Nodejs, but C# with .NET Framework or .NET Core is also a valid option.
This is how it looks. First, initialize the project and install botbuilder package:
|
1 2 |
npm init . npm install botbuilder |
And then, the bot itself:
|
1 2 3 4 5 6 7 8 9 |
const builder = require('botbuilder'); const connector = new builder.ConsoleConnector().listen(); const bot = new builder.UniversalBot( connector, session => session.send(`> Hello world and ${session.message.text}`) ); console.log('> Hello-bot has started'); |
Because it’s going to talk to user via console, we had to create a connector for that (line 2). Then, we create a bot (4), providing it with the connector (5) to a channel and default handler (6) for any conversation that user initiates. In this one our bot simply echoes user messages back, prepending them with Hello world and words.
|
1 2 3 4 |
$ node bot.js > Hello-bot has started I want some cash > Hello world and I want some cash |
Kinds of bot conversations
Even though in the previous example a word “Conversation” has never came up, that line-long arrow function was actually the conversation. In fact, MS Bot Framework has two types of them: waterfall and dialog based.
Waterfall
Waterfall conversation is simply a linear sequence of functions, which guide user through set of questions and answers. For instance, assume we’d want to write a pacifist bot – a guardian of pacifist chat, who asks a set (of one) of questions anyone who wants to join. Without changing previous hello-world example that much, here’s how we could do that.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
const bot = new builder.UniversalBot( connector, [ function (session) { // Step 1 session.send('> Hello and welcome to pacifist chat'); builder.Prompts.choice(session, '> Are you against the war?', 'Yes|No'); }, function (session, results) { // Step 2 if (results.response) { const answer = results.response.entity.toLowerCase(); switch (answer) { case 'yes': session.send('> Oh, so sweet'); break; case 'no': session.send(`> Before I start shooting, I'm kindly giving you a chance to reconsider`); break; } } session.endDialog(); } ] ); |
The whole conversation can be put into two steps: a question and a reaction on its answer. The only question the bot is going to ask is “Are you against the war?”. Because that’s a question with limited number of answers, I used builder.Prompts.choice function to make it as hard as possible to answer anything other than yes or no.
Here’s how conversation with the bot could look like.
|
1 2 3 4 5 6 7 8 9 10 |
$ node bot.js # > Pacifist-bot has started # hey # > Hello and welcome to pacifist chat # # > Are you against the war? (1. Yes or 2. No) # 3 # I didn't understand. Please choose an option from the list. (1. Yes or 2. No) # 2 # > Before I start shooting, I'm kindly giving you a chance to reconsider |
What’s cool, depending on Channel and Connector type, Prompts.choice will try to use UI and tools that given Channel provides. For instance, ChatConnector with Bot Emulator attached could’ve rendered that as two buttons, eliminating the need to type anything at all.
Dialog-based conversations
Creating a conversation based on Dialog objects is more powerful, as with this approach bot starts behaving like a state machine. As it moves between its dialogs it any direction, for end user it might feel like a non-linear conversation.
Dialog itself is an object with miniature waterfall conversation in it. Dialogs can start each other, or stay idle, waiting to particular keywords in ongoing conversation and kicking in when they notice something relevant.
Let’s get back to the original reason why I started to think about writing a bot. If my bot is going to help me with monitoring unreliable tests, these are the kinds of dialogs I might need to have with it:
- See the list of commands that bot understands.
- Tell it to start monitoring unreliable tests.
- Configure a threshold – how often a test is allowed to fail before it’s considered unreliable.
- Tell it to stop monitoring tests.
- Check current status.
What’s interesting, in some scenarios dialog #2 will automatically lead to dialog #3. After all, it’s hard to start monitoring for unreliable tests without being configured first. Another interesting fact is that this bot implies some sort of a state. Firstly, a threshold (dialog 3), which can be associated with current user, and secondly, whether or not monitoring is running, which we can associate with the conversation itself.
This is how we can do that.
Default conversation handler
Let’s add default conversation handler to kick in when I type messages that bot does not understand. Some sort of a help message in response would be enough.
|
1 2 3 4 5 6 7 |
const inMemoryStorage = new builder.MemoryBotStorage(); const welcomeMessage = "Hello! I'm Judge-bot. Please say 'status' to get current status, 'watch' to start monitoring for unreliable tests, or 'stop' to stop it."; const bot = new builder.UniversalBot( connector, session => session.send(welcomeMessage) ).set('storage', inMemoryStorage); |
There’s nothing fancy here except for setting a storage where we’ll keep the state. It’s volatile and will be destroyed when bot exits, but for now it’s good enough.
|
1 2 3 4 |
$ node bot.js # > Judge-bot has started # Anyone? # > Hello! I'm Judge-bot. Please say 'status' to get current status, 'watch' to start monitoring for unreliable tests, or 'stop' to stop it. |
Handling ‘status’ command
What should we do if user types status as we suggested? Start a dialog!
|
1 2 3 4 5 6 7 8 9 10 11 |
bot.dialog('StatusDialog', [ function (session) { const isConfigured = !!session.userData.failuresPercentageThreshold; const isStarted = !!session.conversationData.isStarted; session.send(`> This is what's happening at the moment: Unit test watcher is ${isConfigured ? '' : 'not'} configured Unit test watcher is ${isStarted ? '' : 'not'} started `); session.endDialog(); } ]).triggerAction({ matches: /^status$/i }); |
As you can see, this dialog looks like a miniature version of waterfall conversation, where the only interesting things are reading the values from userData and conversationData and specifying when dialog should start.
Checking:
|
1 2 3 4 5 6 |
$ node bot.js # > Judge-bot has started # Status # > This is what's happening at the moment: # Unit test watcher is not configured # Unit test watcher is not started |
‘Stop’ dialog
This one is pretty straightforward. If conversationData‘s flag is not set – do nothing. Otherwise – turn it off. In real implementation it also would stop some background processes.
|
1 2 3 4 5 6 7 8 9 |
bot.dialog('StopDialog', function (session) { if (session.conversationData.isStarted) { session.conversationData.isStarted = false; session.send("> OK, from now on I'll be ignoring unreliable tests. All of them."); } else { session.send("> Hmm. I haven't really been doing anything to begin with. Anyway, problem solved."); } session.endDialog(); }).triggerAction({ matches: /^stop$/i}); |
‘Configure’ dialog
This one is a little bit more interesting. I want to ask user to enter a number between 1 and 100, representing percentage of false positives, after which a test is considered unreliable. I don’t want to do reading/parsing myself, so let’s allow builder.Prompts.number to handle that.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
bot.dialog('ConfigureDialog', [ function (session) { builder.Prompts.number(session, "How often a test should fail before I treat it as unreliable? (1..100%)", { minValue: 1, maxValue: 100 }); }, function (session, results) { if (results.response) { session.send(`> Got it. Setting the threshold to ${results.response}%`); session.userData.failuresPercentageThreshold = results.response; } session.endDialog(); } ]).triggerAction({ matches: /^configure$/ }); |
This dialog also has a trigger keyword, so I can start it any time I want. But it’s more interesting to allow another dialog to do that.
‘Watch’ dialog
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
bot.dialog('WatchDialog', [ function (session) { if (session.conversationData.isStarted) { session.send("> No worries, I'm already on in"); session.endDialog(); } else if (!session.userData.failuresPercentageThreshold) { session.send("> I'm sorry, it looks like this is the first time you asked me to do that. Let me ask you a few questions first."); session.beginDialog("ConfigureDialog"); } }, function (session) { session.conversationData.isStarted = true; session.send("> Starting to monitor unreliable tests"); session.endDialog(); } ]).triggerAction({ matches: /^watch/i }); |
Here we instruct the bot to end conversation if isStarted flag is already set, or begin a new dialog if configuration is needed. Then, when child dialog finishes, WatchDialog dialog regains the control and continues to the second step.
Having a conversation
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
$ node bot.js #> Judge-bot has started #Hey #> Hello! I'm Judge-bot. Please say 'status' to get current status, 'watch' to start monitoring for unreliable tests, or 'stop' to stop it. #Status #> This is what's happening at the moment: # Unit test watcher is not configured # Unit test watcher is not started # #Stop #> Hmm. I haven't really been doing anything to begin with. Anyway, problem solved. #Watch #> I'm sorry, it looks like this is the first time you asked me to do that. Let me ask you a few questions first. # #How often a test should fail before I treat it as unreliable? (1..100%) #-100 #The number you entered was outside the allowed range of 1 to 100. Please enter a valid number. #50 #> Got it. Setting the threshold to 50% # #> Starting to monitor unreliable tests #Status #> This is what's happening at the moment: # Unit test watcher is configured # Unit test watcher is started |
Pretty cool.
As a bonus, here’s one more trick. Without any single change in conversation flow, I can replace ConsoleConnector with ChatConnector, add small web server to handle HTTP calls and use Bot Framework Emulator to have a nice chat with my bot.
Conclusion
Apparently, writing bots is fun. The biggest thing I haven’t touched yet is connecting a bot to Microsoft Cognitive Services, which would allow me to do some pretty cool stuff.
For instance, you noticed that dialog triggers are pretty rigid, right? If I type Start instead of Watch, corresponding dialog won’t kick in, even though it’s obvious in a given context what the intent was. As a solution, I could’ve connected user intents recognizing module to LUIS – MS service for understanding the language. I saw few examples of how it’s used, and that’s really impressive. For instance, LUIS was able to guess that ‘Hello’, ‘Howdy’, ‘Hey’ and ‘Wazzup!’ probably have the same intent. Or ’10s’ and ‘ten seconds’ are equivalents. Or ‘yesterday’ is a date. And so forth. Wow.
Anyway, I’m still thinking whether or not I should write that bot for unreliable tests, but at this point the bot itself is not a challenge anymore. The only difficult part would be writing connectors for bug tracker, git client and unit test results storage, so in addition to talking bot could actually do something.
