Steps to train Spotter
Plan for training Spotter
In order to effectively train Spotter, you must first start by building a good understanding of who the end users are and what their data requirements are. Let’s take an example of the Salesforce dataset of a Software as a Service company to understand how you should plan for training Spotter.
Step 1: Identify the target users
As discussed in the Data modeling article, you have to start the training process by identifying who will be the target users for Spotter, then listing the types of queries those users will ask.
The Salesforce dataset can be useful for many different teams in your organization. For example, the table below provides a quick summary of the different teams in the organization, and the various user personas and the corresponding use-cases these users may be interested in.
| Team | Persona | Use-cases |
|---|---|---|
Sales |
Sales Leadership (CRO & Directors) |
Sales Pipeline & Performance Analysis Regional Performance Metrics Team Performance & Productivity Customer Segmentation & Portfolio Analysis Sales Process Efficiency |
Customer Success |
Customer Success managers |
Customer Relationship Metrics Product adoption & Usage |
Marketing |
Marketing team |
Marketing Attribution Marketing channel analysis Marketing Spend & ROI Analysis |
Narrowing down to a specific persona and the use-cases you want to cater to for these personas is extremely important during the initial stages. You can prioritize the persona and use-cases based on following factors :
- Demand for insights
-
The teams that are most underserved may also be a good candidate for Spotter to start with as these are the teams that are ready to take data-driven decisions and are mainly blocked by ability to get the insights in a timely manner. Once self-serve analytics are available, these teams will be inclined to use it.
- Value delivered by enabling self-serve analytics
-
Another important factor to consider is the value generated by the organization once their analytics needs are served. For example, in the table above, if you think that the Sales team will be able to close more deals if they get the insights faster, then you should focus on the Sales team.
- Data available for analysis
-
The availability of data for conducting the analysis is the last factor which plays a critical role in deciding the target users and the use-cases.
For the purpose of this example, a quick glance at the Sample Salesforce dataset reveals that the persona we should target is Sales Leadership, and the use-cases we can target are Sales Pipeline & Performance Analysis.
Step 2: Identify the user needs
Once you have identified the target user and the important use cases, the next step is to create a list of questions these users will be interested in. We recommend you to identify the various question groups and then list down a few questions users may have for each of these question groups. For example, the table below provides a quick starting point.
In order to estimate the amount of training required for Spotter please make sure that you are doing a good job of ensuring the following things : Question Groups are exhaustive to cater all the different variations of the questions users may ask. Questions selected under each question group cover the various ways in which business users may end up asking the questions to Spotter. It is not necessary to create an exhaustive list of questions for each group, however, we recommend that you try to cover the different variations users may use while asking the questions.
Once you have identified the list of questions, you can use the guidelines listed above for reference questions and business terms to identify the key business terms and reference questions you should add. Conducting this analysis will help you answer questions like, what training should I provide to Spotter, and how much training is needed for my use case.
Step 3: Train Spotter for the identified users and needs
Let’s use a sample question to understand the training process for Spotter.
Question: What is the revenue generated this quarter?
You can start using the Add Coaching option on top of the Reference question page to start the training. Select the (Sample) Salesforce data source and add this question into the training experience.
Once you have entered the question, press Next. Once you press Next, we generate the answer for this question. The answer generated by AI looks reasonable, but in order for an amount to qualify as revenue, we have to apply a filter on “Status = booked”.
We can modify the query to add the required filter. Once it’s done, we can click on Done to save this question as a reference question.
After the reference question and its answer are submitted, we can proceed to the next step for reviewing the business terms.
The system has suggested three business terms. The first and last business terms seem meaningful and will have the same meaning when used in different contexts, hence these are ideal candidates for business terms. However, we can see that the system has not picked up the filter for “Status = booked” in both these cases. Hence, we should use the X option to add these business terms. Selecting X ensures that the business terms are added in the business terms list but they are not available for use yet.
We can now navigate to the business term section and identify “the revenue” and click on it to open it for editing. In this experience, we can change the business term to “revenue” instead of “the revenue” and modify the mapped search tokens to include the filter on “Status=booked”. After making these changes, we can click on Submit to save the business term with the modified definition.
|
Training Spotter for complex questions
Let’s see the approach to train Spotter for more complex business terms like “Win rate”. The “win rate” is defined as the ratio of the booked deals by the total number of deals. However, while calculating the total numbers of deals, we have to exclude the open deals.
Keep the following guidelines in mind while training Spotter in such cases :
- Complex computations can be broken into a combination of simple computations
-
Create complex formulas using smaller, simpler ones with very clear names while training/fixing the AI-generated answer. This way, LLMs will generalize them better to similar queries.
- Provide human-friendly names for the formula
-
Create formulas with clear names so that the LLM understands the computation process and it can generalize these computations in various scenarios.
For example, to compute win rate, you can create the following three formulas :
- Total concluded deals
-
Count the total number of deals whose status is not Open.
- Deals won
-
Count the number of deals won.
- Win rate
-
Deals won divided by Total concluded deals.
Once these formula definitions are defined, you can correct the generated answer for the whole query and then submit.
Then on the Review business terms screen, accept the definition of win rate.
Once you have completed this exercise, Spotter has learned how to compute the win rate. Try a few of the other sample questions to validate that the training provided to Spotter is working for the remaining set of queries.
What should I do if the training provided doesn’t work for similar queries?
In case you come across scenarios where you are trying to define a specific business term but Spotter is not applying the same to other business terms, don’t try to train Spotter for more queries. In general, Spotter is sensitive to the training provided and one to two examples should be sufficient to train Spotter on a specific concept. However, if after providing one to two examples, Spotter is still not applying the training to other queries, try checking the following things:
- Reference questions are exactly applied to the same query only
-
If the question framing is changed slightly then the query feedback isn’t applied as is. However, it may still be used as a template for answering the question. In such cases, the visualization setting may not get applied and the answer generated by Spotter may be different as well if the interpretation of the new question can be different from the one used for training.
- Business terms are only applied if the exact same term is used in the question
-
Business terms are only selected when the exact same spelling is used in the questions asked. This means that if you have trained Spotter for “win rate” it may not apply to “win frequency” ,”win ratio”, “winning-rate”, etc..
In cases where you are unable to train Spotter after providing one to two examples, please seek help from our experts instead of adding more training examples for the same thing.
Step 4: Validate the training
We recommend two steps for validating the training provided:
- Validate with a set of test queries
-
Make sure at step 2, you are also identifying a set of questions from each question group which can be used to test how well Spotter is working for the use case. These questions can be used for evaluating the accuracy of Spotter after providing new training to Spotter.
- Validate with early adopters
-
Identify a few early adopters of Spotter from the target user group and try to work with them to test the accuracy of Spotter. More often than not, you will uncover new questions from these users that can then be added to the questions identified in step 2. A couple of iterations with these users will help you make the training more robust before rolling out Spotter to all the users in the target audience.
