Behavioral Simulation | The Accuracy Story

A long time back, when we built our first neuroscience based decision engine - we got an accuracy of 70% in simulating the behavior of users. We were greatly disappointed and shelved the product. Only a few weeks later did we realize that most other behavioral tests existing in the market hover between 20% to 40% in their predictive accuracies.

We went back to the drawing board and started working on improving our framework. Almost an year later, we released our first beta application for public testing. In 5 days, we had 534 users create their virtual profile and our decision engine had made 1207 predictions about their behavior. These were specific predictions like given 4 incentives which one would they choose and so on. This specificity of our application itself made us strikingly different from psychometric tests which describe you in broad terms like introvert, conscientious etc.

We also knew of something called The Forer Effect which results in psychology tests appearing to have higher accuracies than they actually have. This effect makes people agree to whatever is shown to them as a psychological result since the predictions themselves are vague and the user has not much idea about it before hand. We wanted to eliminate the Forer effect at all cost. So along with making the predictions specific, we also showed the options to the users before we showed them the predictions so that they have already chosen their option even before they see the predictions.

Result - a whopping 86% accuracy of our decision engine. 1042 predictions were termed as good by the users. However, this still left 2 questions unanswered: corporate validity of these accuracies, authenticity of user base. We then ran several rounds of testing in different companies - all of which resulted in an overall accuracy of 80 - 91% accuracies. In one of the companies - MeritTrac Services Private Limited, 28 employees created their virtual profiles and were analyzed on 172 predictions on their corporate preferences and performance. Result - 79.65% accuracy.

These tests not only put us far ahead of any other traditional methods of assessment in terms of accuracy but also established the universality of our profiles.i.e. once we map an individual's virtual profile, we need not ask the user time and again to retake different tests everytime we want to know something about the person. We can use the same comprehensive profile to simulate a different analysis with the same accuracy.

Usability Study

One of the things we take pride in is the level of efforts we put in to ensure that every single bit of technology we create is highly intuitive & easy-to-use, even for the least computer-friendly user. We like to build every technology solution as a blend of art & technology rather than just a computer based version of existing practices.

When we implemented one such solution with Careernet Consulting, we decided to give the users an option to volunteer their opinions & comments (thanks to Kampyle!) while using our activity based assessment tools. By the end of the project, 60% of the users had volunteered to share their opinions & ideas with us. What struck us, is the sense of excitement that was a part of a majority of those comments

74% of the comments were compliments for the usability & design of the tests - which continues to motivate us as we try to raise the bar higher with each successive product that we design.