Past Experience Q&A For Interview

4 war stories with deep-dive Q&A. Use for behavioral + technical rounds.

→ Related: [[Interview Prep/Coach & Tracking/Projects Done]], [[Interview Prep/Coach & Tracking/Job Switch Hub]]

---

The Project: TGE Logistics Real-Time Dispatch

• The Tech Stack: React (Frontend), Python FastAPI (Backend), MySQL (Persistence), WebSockets (Aqchatter Event Bus), Google Maps API (Visualization).
• The Problem: Tracking 250+ concurrent drivers with high-frequency (5s) telemetry from TED IoT devices.
• The Solution: A hybrid "Hydration + Stream" model where the UI is initialized from MySQL and updated via WebSockets.

Core Performance Hooks (Mention these!)

1. Reconciliation: Updating state via Key-Value objects ($O(1)$ lookup) instead of arrays ($O(N)$ lookup).
2. State Batching: Using a useRef buffer to group WebSocket updates. This reduces CPU usage by 70% compared to direct state updates.
3. Component Memoization: Using React.memo to ensure only moving pins re-render, keeping map interactions smooth at 60FPS.
4. Stale Data Handling: Implementing a client-side heartbeat check to flag "ghost" drivers who stop sending data.

Scaling Strategy (The "FAANG" Answer)

• Ingestion: If scaling to 50k drivers, introduce Kafka as a buffer between the TED devices and the database to prevent write contention.
• Live Cache: Use Redis Geospatial indexes to serve "Live Dispatch" queries, keeping the heavy MySQL instance for historical Audit Reports and Manifest Generation.
• Edge Computing: Use regional WebSocket clusters to reduce latency for global fleets (Global Expansion).

TGE Experience

1. Explain about the Real time system to show driver location
   • I have implemented Google Maps API to show the 250 drivers for give state with their locations updated in realtime using the Webscoket connection which follows the Hybrid Ingestion Pattern(Reconciliation Code) The life cycle contains 3 phases:
     • Initial State Hydration: On component mount, the React app performs a REST API fetch to retrieve the 'Last Known Location' and current status (e.g., active, on break) for 250+ online drivers from the MySQL database.
     • WebSocket Handshake: Once the initial state is rendered, the app establishes a persistent WebSocket connection to the Aqchatter event bus.
     • Real-Time State Reconciliation: As TED devices transmit telemetry every 5 seconds, Aqchatter broadcasts these events to the frontend. I designed the React state management to perform targeted updates—instead of re-rendering the entire map, I update only the specific driver object in the state array, which triggers a smooth position update of that specific pin on the Google Maps layer."

[!IMP] Mention that you "batch" or "throttle" UI updates so the map only refreshes its pins once every 500ms, even if the data comes in faster. This saves CPU and battery for the operator.

2. How do you prevent performance lag when updating 250 pins every 5 seconds?
   • To maintain performance, I used React.memo for individual Marker components to avoid re-renders of entire Map. Additionally, I ensured that I only update the specific coordinate data, keeping the map tiles static.
3. What happens if a driver’s GPS sends 'noisy' data and the pin jumps 5km away suddenly?
   • That is a common telemetry issue. If I were to expand this, I would implement a client-side validation layer. I would check the distance between the 'Last Known' and the 'New' location. If the implied speed is physically impossible (e.g., 500km/h), I would discard that packet as noise or smooth the transition using interpolation (like a sliding window average) to prevent the pin from jumping.
4. How do you handle the 'Reconnection Gap' if the user's internet flickers?
   • WebSockets are stateful, so a disconnect means missed updates. I implemented a Reconnection Strategy: when the socket reconnects, the app triggers a partial re-fetch of the 'last known' locations from the database to reconcile any changes that occurred while the user was offline.
5. Code for the Reconcilliation Pattern to handle the 250+ drivers.

import React, { useState, useEffect, useRef, memo } from 'react'; 
import { GoogleMap, Marker } from '@react-google-maps/api';

// --- Optimized Marker Component ---
// React.memo ensures this pin ONLY re-renders if its position or status changes.
const OptimizedMarker = memo(({ driver }) => {
  return (
    <Marker 
      position={driver.location} 
      icon={driver.isStale ? "gray-pin.png" : "active-pin.png"}
      title={Driver: ${driver.id}}
    />
  );
}, (prev, next) => {
  return prev.driver.location.lat === next.driver.location.lat && 
		  prev.driver.location.lng === next.driver.location.lng &&
         prev.driver.isStale === next.driver.isStale;
});

// High-level Logic for Real-Time Map Updates
const LiveMap = () => {
  const [drivers, setDrivers] = useState({}); // Key-value for O(1) lookups
  const bufferRef = useRef({}); // Temporary storage for "batched" updates

  
  useEffect(() => {
	// 1. Initial Hydration from MySQL
    fetchLastKnownLocations().then(data => {
      const driverMap = data.reduce((acc, d) => ({ ...acc, [d.id]: d }), {});
      setDrivers(driverMap);
    });
    
    // 2. State Batching Interval (Flush updates every 500ms)
	// This prevents 250 individual re-renders per second.
    const batchInterval = setInterval(() => {
      if (Object.keys(bufferRef.current).length === 0) return;
      
      setDrivers(prev => ({
        ...prev,
        ...bufferRef.current
      }));
      bufferRef.current = {}; // Clear buffer after flush
    }, 500);  
	    
	// 3. Heartbeat/Stale Data Check (Run every 10s)
    const staleCheckInterval = setInterval(() => {
      const now = Date.now();
      setDrivers(prev => {
        const updated = { ...prev };
        Object.keys(updated).forEach(id => {
          // If no update for 30s, mark as stale
          if (now - updated[id].lastSeen > 30000) {
            updated[id].isStale = true;
          }
        });
        return updated;
      });
    }, 10000);
    
    return () => {
	    clearInterval(batchInterval);
	    clearInterval(staleCheckInterval);
    }
}, []);

  // 4. WebSocket Stream Ingestion
  useEffect(() => {
    const socket = connectToAqchatter();
    
    socket.on("driver_update", (data) => {
      // Race condition check: Only buffer if it's the newest data
      const currentData = bufferRef.current[data.id];
      if (!currentData || data.timestamp > currentData.timestamp) {
          bufferRef.current[data.id] = { 
            ...data, 
            lastSeen: Date.now(), 
            isStale: false 
          };
      }
    });

    return () => socket.close();
  }, []);

  return (
    <GoogleMap>
      {Object.values(drivers).map(driver => (
        <OptimizedMarker key={driver.id} position={driver.location} />
      ))}
    </GoogleMap>
  );
};

• When you share this, emphasize the "Efficiency of State".
• Initial Hydration: Explain that you first populate the UI with the "Last Known Location" from the database to ensure the operator isn't looking at a blank map while waiting for the first 5-second pulse.
• Targeted State Updates: Mention that you use a Key-Value Pair (Object/Map) for the driver state instead of an Array. This allows you to find and update a specific driver in $O(1)$ time rather than $O(N)$, which is critical when you have hundreds of drivers.
• Component Optimization: Tell them you used React.memo or a similar technique for the Map Pins so that when Driver A moves, Driver B’s pin component does not re-render.

6. How do you prevent "Race Conditions" in the UI?
   • The Problem: Multiple updates for the same driver arriving out of order.
   • The Solution: Each telemetry packet includes a server-side timestamp. In the reconciliation logic, I check if the incoming timestamp is newer than the lastSeen timestamp in my state. If it's older (delayed packet), I discard it to prevent the pin from "jumping backward."
7. What happens if a dispatcher has a slow internet connection?
   • The Problem: A slow client can't process messages as fast as the server sends them, causing a massive backlog.
   • The Solution: This is why I used a Ref-based Buffer. Even if the WebSocket fires 100 times, my setInterval only flushes to the UI every 500ms. If the buffer grows too large, I implement LIFO (Last-In-First-Out)—I only keep the most recent location for each driver in the buffer, dropping intermediate stale updates.
8. How do you test a real-time WebSocket system?
   • The Logic: I used Jest with jest.useFakeTimers() to verify the batching logic. I would "mock" 50 incoming socket events and then fast-forward time to ensure setDrivers was called exactly once with the consolidated data.

---

The Project: ML-Assisted Thresholding (Splunk ITSI)

• The Tech Stack: React (Frontend), Python/Django (Backend), Splunk Enterprise SDK, Scikit-learn (ML Models).
• The Problem: Traditional IT thresholding was manual and error-prone. Users needed to configure thresholds for multiple KPIs simultaneously based on historical patterns without freezing the UI or overwhelming the backend.
• The Solution: An asynchronous recommendation engine that integrated ML models with UI workflows, allowing for bulk threshold adjustments for up to 10 KPIs at a time.

---

Core Performance Hooks (Mention these!)

1. Asynchronous Inference: Separating the heavy ML calculation from the UI thread to prevent browser "freezing" during complex recommendations.
2. Bulk Processing: Designing the API to handle concurrent requests for 10+ KPIs, optimizing the data payload to reduce round-trip latency.
3. Data Recalculation Optimization: Achieving a 90% reduction in search time by refactoring database calls and optimizing object processing patterns.
4. Feature Leadership: Managing the end-to-end lifecycle of a feature that was recognized as a top ML-driven innovation at Splunk.conf.

---

ML Integration Deep-Dive (Interview Q&A)

1. How did you handle ML model latency in a user-facing application?

• The Problem: ML model inference can take several seconds, which is too slow for a standard synchronous REST API call.
• The Solution: I implemented an Async Job Pattern. When the user clicks "Recommend," the frontend triggers a background task and receives a "Job ID." The UI then shows a loading state while the backend processes the ML logic. Once complete, the results are fetched and applied to the KPI configuration screen.

2. How did you optimize the "90% reduction" in search time?

• The Problem: The existing recalculation search was inefficient, making too many individual database calls and processing objects one by one.
• The Solution: I performed a bottleneck analysis and refactored the search to use Bulk Processing. I optimized the SQL-like queries to fetch only the necessary fields and implemented efficient object serialization, which significantly lowered CPU and I/O pressure on the indexers.

3. How did you handle "Data Drift" or inaccurate recommendations?

• The Problem: ML models based on old historical data can give "stale" or irrelevant recommendations if the environment changes.
• The Solution: I built the Entity Adaptive Thresholding feature, which included a nightly recalculation mechanism. This ensured that the recommendation policies were dynamic and adjusted automatically based on the most recent historical data trends.

4. How did you manage the "User Experience" for complex ML logic?

• The Problem: Users often don't trust "black box" ML recommendations.
• The Solution: I architected the frontend to show "ML Confidence Levels" and allowed users to preview the recommendation against historical charts before applying it. This "Human-in-the-loop" design increased user trust and adoption.

5. How did you coordinate with Data Science teams as a Software Engineer?

• The Logic: I acted as the bridge. The Data Science team focused on model accuracy in Python, while I focused on the API contracts, data serialization, and UI responsiveness. We established strict JSON schemas for the model inputs and outputs to ensure that model updates wouldn't break the frontend.

---

Comparison: TGE vs. Splunk Projects

---

The Project: AskTGE — Enterprise AI Intelligence Platform (Multiagent Backend)

• The Tech Stack: Python FastAPI + LangGraph (Backend), AWS Bedrock (Claude Sonnet), Flask + SocketIO (Frontend Gateway), AWS ECS + DynamoDB + S3, SSE streaming, LangChain AWS.
• The Problem: A production multiagent LLM system serving enterprise logistics users had significant latency and cost issues — redundant LLM token processing on every turn, O(N²) streaming bandwidth growth, sequential tool execution, event loop blocking during S3 calls, and 700ms+ cold-start on every ECS container restart.
• The Solution: A systematic 13-issue performance audit and fix across the full stack — from SSE streaming bandwidth to AWS Bedrock prompt caching to async presigned URL generation — reducing per-request latency, cutting token costs by ~90% on follow-up turns, and eliminating cold-start overhead.

---

Core Performance Hooks (Mention these!)

1. Bedrock Prompt Caching: Injected cachePoint markers into the system prompt and tool definitions using langchain-aws native passthrough — subsequent turns skip re-processing ~3,100 static tokens and all associated guardrail scans, cutting input token cost by ~90% per follow-up message.
2. O(N²) → O(N) SSE Streaming: chat_service.py was re-sending the full accumulated response on every streaming chunk — fixed with a last_sent_len delta tracker, reducing bandwidth by up to 50×.
3. Parallel Tool Execution: Replaced LangGraph's sequential ToolNode with asyncio.gather — two concurrent KB searches now run in parallel instead of serially, halving multi-tool response time.
4. Async Presigned URL Generation: S3 boto3 calls were running synchronously inside the astream_events loop, blocking the event loop and stalling token delivery — moved to a post-stream batch using asyncio.to_thread + asyncio.gather.
5. Lazy LLM + Graph Initialisation: LLM and LangGraph compilation happened at module import — moved to lazy getters, eliminating 700ms–1.5s ECS cold-start overhead on every container restart.

---

AI/LLM Performance Deep-Dive (Interview Q&A)

1. How did you reduce LLM costs in production?

   • The Problem: Every conversation turn was re-sending and re-processing the full system prompt (~857 tokens) and all tool definitions (~1,300 tokens) to Bedrock, including re-running all guardrail scans on that static content.
   • The Solution: I implemented Bedrock Prompt Caching by injecting cachePoint markers — one after the system prompt, one after the tool definitions list. After turn 1, Bedrock snapshots the KV cache. Subsequent turns skip re-processing those ~3,100 static tokens entirely, including the associated guardrail scans. This reduced input token cost by ~90% on every follow-up message in a session.
   • Technical detail: Used langchain-aws==0.2.31's native _is_cache_point() support — no subclassing needed. System prompt cache via list-format SystemMessage content; tool cache via appending {"cachePoint": {"type": "default"}} to bind_tools(). Pinned the library version to prevent silent breakage on future releases.

2. How did you fix the streaming performance regression?

   • The Problem: The SSE middleware (chat_service.py) was forwarding the full accumulated_content string on every chunk event. If the response was 1,000 tokens, token 500 would send 500 characters, token 501 would send 501 characters — O(N²) bandwidth growth and a full DOM re-render on every token in the browser.
   • The Solution: Added a last_sent_len offset variable. On each chunk, I compute delta = accumulated[last_sent_len:], yield only the delta, then update the offset. O(N) bandwidth, and the browser only appends new text instead of replacing the entire message node.

3. How do you handle async patterns with LangGraph streaming?

   • The Problem: LangGraph's astream_events is an async generator — you can't run blocking code inside its loop without stalling the event loop and delaying token delivery to the client.
   • The Solution: I identified two blocking patterns: (1) sequential tool execution, (2) synchronous boto3 S3 calls for presigned URLs. For tools I replaced the default ToolNode with a custom asyncio.gather executor that runs all tool calls concurrently. For S3 I moved URL generation entirely out of the stream loop — raw citations are accumulated during streaming, then all S3 calls are batched with asyncio.to_thread + asyncio.gather after the loop ends.

4. How did you handle LLM tool routing for document analysis?

   • The Problem: When users uploaded a document, the original code hard-blocked all tools — meaning users couldn't ask "compare this contract with TGE's standard carrier liability policy" because KB search was disabled entirely.
   • The Solution: Removed the hard pre-routing block and instead handled it at the prompt level. The system prompt instructs the LLM: "If the document content is already in context, answer directly — only call knowledge_base_search for supplementary TGE-internal information not present in the attachment." This gives nuanced control without brittle routing logic.

5. How do you manage cold starts in a containerised LLM service?

   • The Problem: The FastAPI service on ECS was initialising the Bedrock LLM client and compiling the full LangGraph state machine at module import time — adding 700ms–1.5s to every ECS task cold start.
   • The Solution: Moved both behind lazy getters (_get_llm(), _get_graph()). The first request pays the init cost; all subsequent requests use the cached instances. ECS starts faster, and the overhead is paid once per container lifecycle rather than on every deploy.

6. What's your approach to SSE event design in a streaming AI system?

   • The Lesson I Learned: We had two redundant terminal events — final_response (from the LangGraph event parser) and complete (from the app layer). They existed because final_response was the original authoritative event before async citation enrichment was introduced. After the refactor, complete carried the real presigned URLs but final_response was never cleaned up — causing the frontend citation handler to read stale data without download_url.
   • The Principle: In event-driven streaming systems, every event type should have a single authoritative source. When you refactor the data pipeline (e.g., moving enrichment to post-stream), audit all consumers — the terminal event should be the last thing that fires, after all enrichment is complete.

---

Resume Bullet Points (pick 3–4)

Use these as achievement bullets under HCLTech Senior Developer

• Reduced LLM input token costs by ~90% per follow-up turn by implementing AWS Bedrock prompt caching (cachePoint) across system prompts and tool definitions for a production multiagent AI platform.
• Fixed O(N²) SSE streaming bandwidth regression in the AI gateway layer — response middleware was re-sending the full accumulated text on every token; delta-only forwarding reduced bandwidth by up to 50×.
• Eliminated 700ms–1.5s ECS cold-start overhead by converting eager LLM/LangGraph initialisation at import time to lazy getters, and moved synchronous S3 presigned URL generation out of the async event loop into a post-stream asyncio.gather batch.
• Redesigned parallel tool execution in LangGraph using asyncio.gather, halving response time for multi-tool KB search queries; identified and resolved 13 performance issues across a Python FastAPI + AWS Bedrock + LangGraph production stack.

---

---

The Project: Chat History API — DynamoDB Schema Design + Local Encryption Layer

• The Tech Stack: Python FastAPI (async), DynamoDB Single-Table Design, AWS KMS (legacy path), Fernet symmetric encryption (new path), aioboto3, AWS ECS.
• The Problem (v1 — Encryption): The existing chat history system used AWS KMS for encrypting every message — every read and write required a network round-trip to the KMS endpoint, adding latency to every history fetch. The solution needed to operate independently of KMS without breaking the legacy encryption path.
• The Problem (v2 — Schema): The original DynamoDB schema used random UUID sort keys. This meant the application had to fetch all sessions and sort them in memory on every sidebar load — an O(N) application-side sort with no native DynamoDB ordering. Session metadata updates also had a race condition under concurrent writes where two threads could independently create phantom duplicate sessions.
• The Solution: Two-phase redesign. Phase 1: a local Fernet-based encryption service replacing per-message KMS calls. Phase 2: a new Single-Table Design with epoch millisecond timestamps embedded in Sort Keys for native DynamoDB ordering, and an Atomic Bump Pattern using TransactWriteItems to guarantee ACID-safe session promotion.

---

Core Design Hooks (Mention these!)

1. Zero GSIs: All access patterns served from the base table via begins_with Sort Key prefix queries — no GSI write amplification.
2. Native DynamoDB Sort: Epoch MS timestamps (16-digit zero-padded) in Sort Keys — DynamoDB sorts lexicographically at read time, no application-side ordering.
3. Atomic Bump Pattern: Every LLM response atomically promotes the session to the top of the sidebar via TransactWriteItems (Delete old metadata SK + Put new metadata SK with fresh timestamp).
4. Race-Condition Safety: ConditionExpression: attribute_exists(SK) on the Delete leg — if two threads race, the second transaction fails cleanly instead of creating a phantom session entry.
5. Local Encryption over KMS: Fernet symmetric key stored in LOCAL_CHAT_ENCRYPTION_KEY env var — no network round-trip to KMS per message, reducing per-read latency while maintaining the existing KMS path for legacy compatibility.

---

DynamoDB Design Deep-Dive (Interview Q&A)

1. How did you design the DynamoDB schema for chat history?

   • The Problem: Chat history needs two access patterns: (1) list user's recent sessions sorted newest-first, (2) fetch all messages in a session chronologically. The original UUID sort key design couldn't satisfy either without fetching everything and sorting in application memory.
   • The Solution: Single-Table Design with two record types sharing the same partition key USER#<user_id>:
     • Metadata records: SK = METADATA#AGENT#<agent_id>#TIME#<epoch_ms_16>#SESSION#<session_id> — agent prefix enables begins_with scoped queries; epoch MS in the middle gives native sort.
     • Message records: SK = SESSION#<session_id>#MSG#<epoch_ms_16>#<interaction_id> — session prefix scopes to one conversation; epoch MS gives chronological order.
   • ScanIndexForward=False on metadata queries → newest sessions first. ScanIndexForward=True on message queries → standard chat flow. No GSIs, no in-memory sort.

2. What is the Atomic Bump Pattern and why did you need it?

   • The Problem: Every time a user sends a new message, their session should move to the top of the sidebar. Because the session's recency is encoded in the Sort Key itself (the timestamp), you can't just update a field — you have to replace the entire record with a new SK containing a fresh timestamp. A naive Delete + Put sequence is not atomic: two concurrent writes can both read the old SK and both try to delete it, causing one to fail silently or create orphan records.
   • The Solution: TransactWriteItems with three operations in a single atomic call: (1) Put the new message record, (2) Put the new metadata record with the fresh timestamp SK, (3) Delete the old metadata record with ConditionExpression: attribute_exists(SK). If the old record was already deleted by a concurrent write, the condition fails and the transaction is cancelled — the caller retries with exponential backoff from a fresh pre-flight query.

3. Why did you move from KMS to local Fernet encryption?

   • The Problem: AWS KMS GenerateDataKey and Decrypt calls each add ~5–20ms of network latency. For chat history — which can fetch 50+ messages per sidebar load — this compounds to meaningful delay. KMS also has per-account request quotas that could throttle under load.
   • The Solution: Implemented a LocalEncryptionService using Python's cryptography.fernet library. The symmetric key is a 32-byte base64 value stored as LOCAL_CHAT_ENCRYPTION_KEY in the environment. Encryption/decryption is pure CPU — no network call. The service is completely independent of the existing encryption_service.py (KMS path), so legacy data and the old code path remain untouched — dual-path compatibility.
   • Trade-off to acknowledge: KMS provides automatic key rotation, audit logs via CloudTrail, and hardware-backed security. Local Fernet requires manual key rotation and you lose the KMS audit trail. The trade-off was accepted given the internal user base and the significant latency reduction.

4. How did you handle the schema migration from UUID sort keys to timestamp sort keys?

   • The Problem: Existing production data had random UUID sort keys — you can't rename a DynamoDB Sort Key in place, and you can't have downtime to do a hard cutover.
   • The Solution: Zero-downtime dual-write migration. New code writes to both the old UUID-based SK and the new timestamp-based SK simultaneously. A backfill script rewrites all existing records to the new format in staging, validated against sort order before production. Once the new schema is confirmed stable, the old write path is removed.

5. What trade-offs did you make in the DynamoDB design?

   • Partition co-location: Both metadata and message records share the same partition USER#<user_id>. This is intentional at current scale (~300 records) — DynamoDB's 10GB partition limit and throughput ceiling are nowhere near stress. The documented v2 migration path is: move messages to PK = SESSION#<session_id> when any single user exceeds ~10k sessions or shows throttling.
   • Pre-flight query cost: The Bump Pattern requires a Limit=1 query to find the current metadata SK before every write. Accepted at current volume — optimisation path is Redis/ElastiCache keyed by session_id when volume grows.
   • All-agents query removed: The original design had a cross-agent session list requiring in-memory sort. This was explicitly cut — agent-scoped queries only, which begins_with handles natively.

6. How do you ensure ACID compliance in a NoSQL database?

   • DynamoDB's TransactWriteItems provides ACID guarantees across up to 100 items in a single transaction. The key is using ConditionExpression as the safety net — it turns a logical "check-then-act" into a single atomic operation at the storage layer. Without the condition, the Delete is unconditional and two concurrent threads can both succeed, creating inconsistent state. With the condition, only one succeeds; the other gets TransactionCanceledException and retries cleanly.

---

Resume Bullet Points — HistoryAPI (pick 2–3)

Add under HCLTech Senior Developer alongside AskTGE bullets

• Designed and implemented a DynamoDB Single-Table schema for enterprise chat history with zero GSIs, native sort ordering via 16-digit epoch millisecond Sort Keys, and an Atomic Bump Pattern (TransactWriteItems + ConditionExpression) to guarantee race-condition-safe session promotion under concurrent writes.
• Replaced AWS KMS-per-message encryption with a local Fernet symmetric encryption service, eliminating per-read network round-trips to KMS and reducing chat history fetch latency — legacy KMS path preserved for dual compatibility.
• Authored a principal-architect-reviewed Technical Design Document (v3) for production DynamoDB schema migration including zero-downtime dual-write backfill strategy and documented partition hot-spot mitigation triggers.

---

Comparison: All Projects