Portrait of Zhuohao Ni

Zhuohao Ni

I am an M.S.E. student in Robotics at the University of Pennsylvania. Previously, I studied Computer Engineering at the University of British Columbia.

I am broadly interested in AI systems for multimodal and embodied intelligence. My work spans foundation-model infrastructure, multimodal learning, and robot learning.

Publications

MLAN project mark

Towards KnowFM @ ACL

MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models

Jianhong Tu*, Zhuohao Ni*, Nicholas Crispino, Zihao Yu, Michael Bendersky, Beliz Gunel, Ruoxi Jia, Xin Liu, Lingjuan Lyu, Dawn Song, Chenguang Wang

Introduces a text-heavy visual instruction-tuning strategy that transfers instruction-following ability and domain knowledge across modalities, matching vision-heavy mixtures across 12 benchmarks with as little as half the training tokens.

FiLMed architecture aligning localized visual features with fine-grained clinical concepts

Actionable Interpretability Workshop @ ICML

FiLMed: Fine-Grained Visual Tokens Align with Localized Semantics

Zhuohao Ni, Xiaoxiao Li

Introduces a fine-grained concept-bottleneck framework that aligns learnable attribute tokens with localized visual evidence, enabling spatially grounded and clinically interpretable skin-lesion diagnosis.

Community-aware brain connectome transformer architecture

Preprint · arXiv

Learnable Community-Aware Transformer for Brain Connectome Analysis with Token Clustering

Yanting Yang*, Beidi Zhao*, Zhuohao Ni*, Yize Zhao, Xiaoxiao Li

Brings learnable community discovery to connectomic analysis by coupling transformer representations with token clustering, replacing fixed brain partitions with data-driven functional communities.

* Equal contribution

Projects

AI Systems

Microsoft · Azure AI Inference Platform

Prompt Cache Advisor

Built an AI infrastructure system for Azure OpenAI that improves inference efficiency by diagnosing prompt-cache performance at scale and translating system signals into actionable optimization strategies.

ClickCut project mark

ShotAI

Multimodal Video-Editing Agent

Built ClickCut, a multimodal planner-executor agent that converts creative intent into structured, executable video edits. Orchestrated multiple foundation models through tool calling and replayable workflows, making complex editing tasks reliable, inspectable, and easy to refine.

Walnut project mark

Walnut

AI Learning Platform

Co-founded Walnut, an AI learning platform that helps international students follow lectures through real-time transcription, translation, and context-aware study assistance. Led the technical direction across multimodal note-taking, retrieval over course materials, and the product infrastructure that turns live classroom content into searchable, personalized study workflows.

Robotics

Physical drone racing course used for sim-to-real policy deployment

Penn

Autonomous Drone Racing

Developed a PPO-based autonomous racing policy with gate-aware observations, reward design, and domain randomization for robust high-speed flight. Bridged simulation and physical deployment on Crazyflie with ROS 2, placing 1st out of 39 teams.

Object detections used by the robotic sorting arm

UBC

Automated Robotic Sorting Arm

Built an end-to-end edge robotics system on NVIDIA Jetson Nano, combining YOLO perception, TensorRT inference, and autonomous manipulation for real-time component sorting. Designed modular perception, control, and monitoring layers to support reliable deployment and rapid debugging.

Particle-filter SLAM map and estimated trajectory

Penn

Learning in Robotics

Built a suite of robotics learning systems that connect estimation, mapping, reinforcement learning, neural rendering, and model-predictive control across complete perception-to-action pipelines. Implemented and evaluated these methods in simulation, with an emphasis on how learned policies and classical robotics components interact.

Experience

MicrosoftSoftware Engineering Intern, Azure AI Inference Platform
ShotAICo-Founder / Software Engineer, AI Systems
LEAP A.I.Founding Engineer
WalnutCo-Founder / Tech Lead

Education

University of PennsylvaniaM.S.E. in Robotics
University of British ColumbiaB.A.Sc. in Computer Engineering, Graduated with Distinction