Causal Reinforcement Learning based Agent-Patient Interaction with Clinical Domain Knowledge

Wed, 01 Oct 2025 00:00:00 +0000

Speaking Memories: A Multimodal Adaptive Dialogue Framework for Reminiscence Robotics

Sun, 28 Sep 2025 00:00:00 +0000

This work is driven by the results in my on HRI.

A Distributed Multimodal Robotic Framework for Emotion-Aware Reminiscence Dialogue in Dementia Care

Mon, 01 Sep 2025 00:00:00 +0000

We introduce an embodied robotic implementation of the \textbf{PARTNER} framework (Personalized AI and Robotics to Nurture Engaging Reminiscence), a distributed multimodal architecture for emotion-aware, personalized dialogue in socially assistive contexts. The framework has three components: a secure cloud portal for managing media, a local server for processing multimodal inputs, and an embodied robot client. PARTNER combines auditory, visual, and textual inputs using Whisper for speech transcription and a vision–language model (GPT-4o) that infers implicit affect from facial snapshots and dialogue history, rather than relying on rigid emotion classifiers. To enhance reproducibility and support future model training, PARTNER incorporates a real-time logging pipeline that synchronizes user inputs, sensor streams, and model outputs into a structured dataset. We provide a system-level evaluation on our robot, measuring end-to-end command–response latency, transcription accuracy, and dialogue coherence under varied sensing and environmental conditions. Our experiments show sub-3,s loop latency on our testbed, robust transcription across various noise environments, and consistent responses during multi-turn dialogues, These findings validate PARTNER as a deployable platform for adaptive human–robot interaction. To our knowledge, PARTNER is the first Socially Assistive Robotics (SAR)-oriented system that (i) unifies a cloud portal for reminiscence media with a locally executed interaction server and an embodied agent, (ii) leverages VLM-based implicit affect cues for dialogue policy, and (iii) offers a real-time multimodal logging substrate to facilitate future domain-specific VLM/LLM fine-tuning.

Multimodal Perception-Driven Decision-Making for Human-Robot Interaction: A Survey

Tue, 05 Aug 2025 00:00:00 +0000

Interval Short-Term Traffic Flow Prediction Method Based on CEEMDAN-SE Nosie Reduction and LSTM Optimized by GWO

Wed, 10 Aug 2022 00:00:00 +0000

An example preprint / working paper

Sun, 07 Apr 2019 00:00:00 +0000

This work is driven by the results in my on LLMs.

Note

Create your slides in Markdown - click the Slides button to check out the example.

Add the publication’s full text or supplementary notes here. You can use rich formatting such as including .

Publications | Zachary Zhao

Causal Reinforcement Learning based Agent-Patient Interaction with Clinical Domain Knowledge

Speaking Memories: A Multimodal Adaptive Dialogue Framework for Reminiscence Robotics

A Distributed Multimodal Robotic Framework for Emotion-Aware Reminiscence Dialogue in Dementia Care

Multimodal Perception-Driven Decision-Making for Human-Robot Interaction: A Survey

Interval Short-Term Traffic Flow Prediction Method Based on CEEMDAN-SE Nosie Reduction and LSTM Optimized by GWO

An example preprint / working paper