Step 1: Accepting Multi-Modal User Input
The user experience starts at the interface, but instead of limiting interaction to one mode, our system supports text, voice, image, and manual input, making it as frictionless as possible for users across age groups and tech-savviness levels.
- Voice commands are captured and transcribed using Google Speech-to-Text (STT) or Whisper APIs. A user might say, “What can I eat after skipping lunch?” and the system instantly decodes the intent and context.
- Image inputs, such as a photo of a meal, are processed through a custom image recognition model trained on food datasets. It identifies ingredients, portion sizes, and timestamps for logging.
- Text input allows users to type queries, like “Suggest a low-carb dinner,” parsed for intent using LLMs.
- Manual input is also supported for users who want granular control, such as logging a protein shake with exact macro values.
This flexible input layer ensures that no matter how the user chooses to interact, the system can understand and respond.
Step 2: The Agent – Real-Time Orchestration Hub
All incoming inputs converge at the Agent, the central engine that determines what happens next. It’s responsible for interpreting user queries, extracting context (such as time of day, previous meals, and activity data), and determining which downstream modules to invoke.
The Agent performs:
- Intent mapping: Is the user logging food, seeking suggestions, or asking a health question?
- Decision routing: Should it query the Nutrition API for nutrient values? Should it pass context to the LLM for behavior coaching? Or log new data to the database?
The Agent ensures that all interactions—regardless of complexity—are processed within seconds, delivering the right outcome with minimal delay.
Step 3: Specialized AI Subsystems for Autonomy
Behind the Agent are three critical subsystems that handle domain-specific operations:
A. Logging Task Subsystem
This is where meal entries are transformed into structured, analyzable data.
- NLP is used to interpret entries like “Grilled tofu with brown rice” and decompose them into ingredients.
- Nutrition databases like Nutritionix are queried to fetch macro/micro values.
- Entries are logged against the user’s timeline for historical analysis.
Whether input via image, voice, or text, this subsystem ensures meals are logged with precision and context.
B. Listing Task Subsystem
This module is responsible for personalized food recommendations. It considers:
- User preferences (e.g., gluten-free, vegan)
- Regional constraints (e.g., U.S.-based ingredients)
- Temporal context (e.g., morning vs. evening meals)
- Historical patterns (e.g., recurring nutrient deficiencies)
It returns suggestions tailored to both goals and behavior, adapting in real time as patterns evolve.
C. Health Q&A & Pushback System
When users go off track—or ask broader health questions—this system steps in. It powers:
- Nudges: “You’re 90% to your sodium limit today.”
- Swap suggestions: “Try grilled tempeh instead of fried paneer.”
- Behavioral cues: “You’ve skipped two meals today—consider a balanced high-fibre snack.”
If the system detects ambiguous queries, it defers to the LLM (more on that next).
Step 4: LLM (Gemini/GPT) – The Conversational AI Brain
Large Language Models are integrated to make the app feel intelligent, empathetic, and trustworthy. Using
LangChain and vector databases like
Pinecone, this layer adds memory and reasoning.
It handles:
- Conversational prompts like “What should I eat after a heavy lunch?”
- Educational insights like “Why fiber helps reduce cravings”
- Q&A with embedded medical docs, using RAG pipelines and document retrieval tools like Google Vertex AI
Over time, the LLM builds memory around user preferences, allowing it to deliver increasingly smart and personalized responses.
Step 5: Data Layer and Nutrition APIs
Data flows from the Agent and AI subsystems to a centralized DB Layer, where all meal entries, nutrition info, user context, and session logs are stored securely.
The system uses APIs such as:
- Nutritionix / USDA to retrieve precise nutrient breakdowns
- OpenAI or Gemini for coaching and conversational feedback
- Apple Health or Fitbit to sync physical activity and biometrics
The logging is real-time and timestamped, which allows for rich trend analysis and behavioral tracking.
Step 6: Retrieval-Augmented Generation (RAG) Layer
To support medical accuracy, especially for queries about chronic conditions or dietary limitations, the system integrates an
RAG pipeline.
This includes:
- GCP object bucket storing JSONL-embedded documents
- Google Vertex AI Vector Search for semantic lookups
- Query results are passed back to the LLM for contextually accurate answers
Whether it’s answering “Is quinoa safe for diabetics?” or “Should I avoid dairy if I’m lactose intolerant?”, this module provides grounded responses backed by real health data.
Step 7: Security, Privacy, and HIPAA-Readiness
Handling health data in the U.S. means following strict compliance protocols. The system is designed to be
HIPAA-ready with:
- Token-based authentication
- Encrypted user data
- BAA-ready structure for clinics or insurers
- Role-based access and detailed audit logging
This foundation allows seamless expansion into health-tech partnerships while protecting user privacy.
Step 8: External API Integrations
To function as a true health assistant, the system integrates with:
- Apple Health & Fitbit APIs for biometrics
- Twilio / Firebase Cloud Messaging for nudges and reminders
- Nutrition APIs for dietary information
- OpenAI / Gemini APIs for generative coaching
- Mixpanel or Segment for usage analytics and behavior tracking
Sample Code Prompt (Behind-the-Scenes)
