Converse: Making drug information conversational

Why This Project?

Personal Learning Goals

• I want to gain hands-on experience with building a modern, scalable, and cost-effective LLM-powered web app.
• I also hope to understand the entire pipeline, from data collection to chatbot optimization.

Real-World Inspiration

• A friend working in the public health sector highlighted a major pain point:
  • Extracting data from drug labels is time-consuming since crucial information is buried within long passages of text.

Impact

• Showcased the app to demonstrate RAG capabilities within my team. Sparked curiosity among colleagues, leading to further discussions.
• Showed it to friends in the healthcare & pharmaceutical fields, receiving valuable feedback and feature requests.

Pipelines

1. Authentication Flow

graph TD
    A[Start:User Reach Auth Middleware] --> E
    E{Is sessionId available?} --|No|-->  Q[Authentication Failed: Bounce back to LogIn]
    E -->|Yes| G[Validate session with service]
    G --> H{Is session valid?}
    H -->|Yes| I{Is session fresh?}
    I -->|Yes| J[Create new session cookie]
    J --> K[Set session cookie in cookies]
    K --> L[Cache authentication result]
    I -->|No| M[Create blank session cookie]
    M --> N[Set blank session cookie in cookies]
    H -->|No| Q
    L --> D[Authentication Success]
    N --> D  

2. RAG Pipeline (Improvement In Progress)

  graph TD
    A[Prompt] -- Raw Prompt + Chat History --> B(Rewriter - Rewrite Ambiguous References)
    B -- Refined Prompt --> C(Name Entity Recognition - Extract Subjects)
    C -- Extracted Subjects --> D(Vector Store - Retrieve Top 3 Matches)
    D -- Context + Prompt --> E(LLM)
    E -- Generated Response --> F(User)

Challenges & Solutions

1. Data Collection & Cleaning

• The quality of LLM responses heavily depends on the source material.
• Solution: Automate data collection & preprocessing with Python.

        flowchart TD
            A[Start: Initialize Environment]
            D[Fetch Drugs Detail from FDA]
            E[Unzip/Parse Data]
            K[Null Handling, Feature Selection, Stratified Sampling]
            F[Merge Data into a Single JSON File]
            G[Save JSON File to Mounted Drive]
            H[End]

            A --> D
            D --> E
            E --> K
            K --> F
            F --> G
            G --> H
    

2. Chatbot Losing Context

• Problem:
  • The Meta-Llama-3.1-8B model has an 8k context window, which isn’t sufficient for complex drug-related queries.
  • In a typical drug investigation scenario, this context is quickly exhausted.

Potential Solutions Considered

• Solution 1: Use a High-Context Window Model

  • Options:
    • Meta-Llama-3.1-8B-Instruct-Turbo (130k context)
    • GPT-4o-mini (128k context)
  • Challenges:
    • Cost: 80% - 220% higher (~$0.2 - $0.4 per 1M tokens).
    • Hallucination Risk:
      • Smaller parameter models (8B LLaMA) struggle with chemical/drug names. — Especially beyond 2 turns
      • Confuses prior chat outputs with user queries, leading to misinterpretation.

• Solution 2: Semantic Router (Topic-Based Context Switching)

  • Idea: Classify queries into semantic categories and retrieve context accordingly.
  • Issue: Drug-related queries have nuanced dependencies, making classification-based approaches unreliable.

• Solution 3 (Adopted Approach): Rewriter Agent

  • Use a small language model (SLM) like Google Gemma 2 to rewrite queries with historical context.
  • Advantages:
    • Cost-effective (since SLMs are cheaper & can even run on-device using Transformer.js).
    • Effectively identifies the main focus of the conversation.
  • Limitations:
    • The rewriter model itself has a context window—once exceeded, context loss occurs again.

3. Chat Streaming: Optimizing a Multi-Agent Workflow

• Ensuring real-time responses while handling multiple agents (rewriter, retriever, LLM, reranker) efficiently.

4. Session Management – Next.js Authentication Pitfalls

• Initial Approach: NextAuth.js
  • Too heavy—bundled with unnecessary features.
• Switched to Lucia
  • Pros: Lightweight, provides essential features.
  • Cons:
    • Not reliable in middleware.
    • Poor integration with DynamoDB.

Lessons Learned

• Authentication should happen at the middleware level for security.
  • Frontend-based authentication is bypassable by pausing JavaScript execution.
• Use Redis for chat history caching—it improves performance and reduces costs.
• LLM temperature settings matter:
  • For research-oriented assistants, lower temperature values improve accuracy.

Potential Improvements & Optimizations

1. Balancing Cookie-Based vs. API-Based Validation

• Cookie-Based Validation: Easier to implement, but less secure.
• API-Based Validation: More secure, but adds complexity & cost.

2. Reranking Strategy for Drug Queries

• Problem:
  • Some rarely used drugs are picked over more relevant ones due to high similarity scores.
• Potential Fix:
  • Google Search Index Integration – Rank based on search frequency of drug names.

3. GraphRAG for Pharma Research

• Why?
  • Pharmaceutical companies produce thousands of drugs.
  • A GraphRAG system could:
    • Show drug-manufacturer relationships.
    • Assist pharmaceutical law researchers.
    • Provide business intelligence insights.

4. Improve RAG Pipeline

• Unecessary Actions: Vector Index search always happen, regardless if user made new request.
  • Potential Solution: use a conversational router framework like ARCH.
• Latency: Primarily caused by bouncing between difference LLM providers
  • Potential Solution: integrate with one provider like AWS Bedrock.

Final Thoughts

• This project was a deep dive into LLM-based retrieval, context retention, and chatbot optimization.
• While many challenges arose, creative solutions (like query rewriting) significantly improved accuracy without drastically increasing costs.
• There’s still room for scaling and optimization, particularly with GraphRAG and AWS infrastructure.

I would love to hear your feedback! how would you optimize further? 🚀

Contact Me

[Top]