Re-thinking Insights
Until now, the analytics and insights functions have been shaped up incrementally, and as data has increased, tools have evolved, the way to reach insights has transformed. Insights and analytics is significant for a business simply because it helps the way business makes decisions, and evaluates whether the current strategies are fruitful or requires any intervention. These interventions helps generate a loop - that leads to successful implementation of data & analytics in running a business. Since AI is changing pretty much everything in all spheres of business, the data & analytics function is no different and here is why:
1) Engineering: data needs to be processed before it can be transformed into useful information. This processing is made easier using AI - engineers can now write code easily using copilot, and even generate pipelines much faster using tools such as Databricks Lakeflow designer, Claude Code, Codex, etc. Data Engineering is being made easier especially in relation to the mechanical parts it entails.
2) Data Analysis: data analysis in itself is relatively complex field encompassing descriptive analytics, statistical analysis, machine learning and predictive / prescriptive areas. Data analysis has been contributed usually by models wherein a dedicated team of analysts support multiple stakeholders by answering business-as-usual and adhoc questions. However, AI is shifting the analysts up the value chain and making them think about the problem framing, context, validation and decision intelligence.
The analytical technique and implementation mechanisms can vary and what matters is the robust solution and answer.
3) Insights generation - this is the holy grail because this is ultimately what leader wants to influence. AI has made this bit streamlined as consuming knowledge (structured and unstructured) and providing answers through self-attention mechanisms is what AI is naturally good at.
However, business needs to make fast and good decisions - insights has a strong connection with trust. That is what makes this part one of the most complicated and nuanced bit in analytics.
4) Decision intelligence - trust is what binds a decision with insights. Business needs to trust agents and the way to make them trust is to keep them involved, keep the momentum going, and to act on their feedback continuously and rigorously until we reach a stage where AI is performing up to the mark. Gartner mentions that its the decisions and not the insights ultimately need to be influenced, governed and operationalized.
AI can dramatically shorten the distance between a question and an answer. The real challenge is shortening the distance between an answer and a trusted decision.
Use Case
I have done a small exercise assessing off the shelf Databricks Genie agent to process clinical trials data that is registered on Clinical Trials Transformation Initiative and is freely available. AACT is a publicly available relational database that contains all information (protocol and result data elements) about every study registered in ClinicalTrials.gov. Being in pharma myself, this dataset seems super interesting to me as it delves into the clinical trials spanning across multiple therapeutic areas such as HIV, Oncology, Immunology, etc.
The first step before we start building out an AI agent is to assess the data, process it and make it ready to be ingested into the tool. Lets begin with the first step - understanding the data: this dataset has 2 files (there are hundreds of other files but for this POC, lets consider major 2 datasets):
- Studies - this dataset includes the list of all the clinical trials including their description, start date, end date (or proposed end date), study type, phase, and other metadata fields.
- Sponsors - details of the sponsors of the clinical trials including the fact whether they were leads or collaborators across the study.
Testing without training
The process of using an AI tool without training is called - zero shot learning, which involves testing the tool on pre-trained model. There could be cases wherein the model already performs up to the mark without training - however, for most real world data use cases, complexities involved are such, that the tool does require training pertinent to metadata, calculation logics, business definitions, context and relationships, especially for long run. IBM explains that with the growing amount of data and information, context becomes the heart of making better business decisions.
IBM Watson articulates that the way to get better answers in Business Intelligence from AI is to maintain column semantics, relationship intelligence, calculation logic, operational semantics, and AI instructions. This is what will make all the difference in the world in terms of a reliable decision intelligence tool.
Training my AI tool
The AI tool I utilized during this process is Genie Agent - which is one of the components of AI offered by Databricks in the space of AI/BI. In my own experience, Databricks has improved massively with this offering. Earlier, it used to transform questions -> SQL queries and simply provide an response based on query's result and the performance was a bit average. Genie capabilities have massively improved with its Genie code delving much deeper into the data, and applying multi-step reasoning mechanism to provide the best response along with the reasoning. It showcases not only the SQL query result, but pairs the result with business-friendly insights.
More recent Genie agent experience
Question → Understand context → Identify relevant data → Generate/run analysis → Perform multi-step reasoning → Interpret results → Business insight
There are multiple ways to train a Genie agent with the business context and definitions. I have followed 3 steps in the most relevant order for the Genie agent to be trained on my use case.
Step 1: I added metadata for the tables including:
- what a table constitutes,
- definition of all the major attributes
- synonyms
Step 2: Adding example queries
Adding example queries takes precedence over adding English instructions. So, I added a few example questions and their SQL queries so that the agents knows how to proceed when asked a similar question.
I added 13 such queries by creating key business questions that require answering and then branching out those questions into sub-questions. Number of examples to be added depends on complexity of agent, and it is an iterative process.
Step 3: Adding English instructions
Anything global in nature which the agent should keep in mind, and what has not been added through SQL instructions, should be fed into this category. It comes under General Instructions tab within Instructions.
Example Business Questions
Without further due, lets dig in and ask some interesting questions to the agent related to clinical trials. Note - this is an iterative process and I had to validate the answers and in case of deviation or erroneous results, I had trained the agent iteratively with relevant instructions.
Below screenshots showcase some of the questions I asked the agent and how it responded based on the knowledge fed into it.
Validation & Feedback
There are mechanisms to validate the output, and provide feedback to genie. The thumbs up/thumbs down button across each response helps us provide feedback, and in case of thumbs down, we are able to provide subjective reasoning. Show code feature helps us review the SQL query behind the response, and we are also able to alter the query in case of any deviation from expectations.
Genie lets the space admins review the usage stats, the number of likes and dislikes, the reason for dislikes so that the agent owner can take necessary actions and interventions to make the agent better.
Key Takeaways
Oncology Dominates the Competitive Landscape: Cancer research represents 15% of all trials—twice the volume of any other therapeutic area. AstraZeneca leads in academic/research partnerships for cancer trials
Collaboration Strategies Differ Significantly: Self-directed model: Novartis and Boehringer lead 85%+ of their trials. Partnership model: Merck splits evenly between lead and collaborator roles
Trial Execution Speed Varies by Complexity: Early-phase trials (Phase 1) complete in ~2 years on average. Late-phase trials (Phase 3) take ~3 years—65% longer due to scale and endpoints
4. Data Quality Issues Signal Compliance Risk: Nearly 3% of recent registrations are retrospective—some trials registered 30+ years after completion. Implication: Audit your trial registration practices to ensure regulatory compliance and avoid reputational risk
Conclusion
This experiment reinforced an important point: the quality of an AI analytics agent is not determined simply by the underlying model. It is determined by how well the organization encodes its data, definitions, relationships, business logic and feedback mechanisms into the system.
Each sphere of analysis is being transformed to a high degree by AI - platforms and tools are becoming strong, which is making the process of analysis easier and faster. Organizations are trying their strategies to re-think analytics, and it will take a few permutations and combinations to envision an optimized model and ways of working. One thing I have learnt is that there wouldn't be a one size fits all approach. Data & analytics models will and should be different for different organizations basis their appetite for risk, the stakes at play, and budgets.
References
https://www.gartner.com/en/documents/7864181?utm_source=chatgpt.com