Hack2023 2nd Prize
2nd Prize Award
Project Sheikah Tower
AI Assistants powered by local people
Enhance usefulness based on time and place
Team
from Google LLC and Peking University
- Qi Tang - Senior Hardware Engineer - Google
- Yi Han - Google
- Sharu Jiang - Peking University
Introduction
Edge Native Real-time Voice AI Assistant provides local-info enhanced language and speech services (e.g. real-time translation and AI voice-bot) using the state-of-the-art AI models.
The key features are the real-time streaming services by edge computing. We will also leverage the ETSI MEC APIs to fine-tune or prompt the LLM with available local information (such as dialects, geographics, local culture) to provide faster and more useful user contents. We will demonstrate the advantages of edge computing compared to traditional on-device and cloud based services. The end user interfaces can be mobile devices, wearables, IoTs, robots and/or vehicles.
Main Features:
● Users can easily find the nearby virtual assistants from a map view by leveraging MEC APIs
● Local AI virtual assistant indexed by ZoneID and CellID
● “Local” means the Vector Database and Prompts are location dependent
● The Vector database and Prompts are uploaded and designed by the local business owners;
● The virtual assistants can be sophisticated / the-state-of-art AI models serving as a real-time language interpreter (For example, Meta’s latest Speech-to-Speech Massive Language models) which also can be found by the user from the map (as long as it is within the same ZoneID or CellID)
● The finding range can be also flexible: for example, indoor localization information from MEC APIs which is used serving for a museum exhibit tour (room specific) or a city tour based on the user’s device GPS signal
● The app is agnostic to various user end devices because computation, memory and location information is not on device per se. We choose iOS for demonstration purpose only.
Software resources
• Project repository
https://github.com/Dako2/sheikah-tower.git