#Mixed_Reality #AI_Application #System_Design

Joley Liang

Selected Projects

Mixed Reality Research

Immersive AR Scaffolding for Vocational Training

A HoloLens 2 system utilizing Scaffolding Theory to assist students with intellectual disabilities in mastering multi-step vocational tasks.

Unity 2020.3 MRTK Vuforia Engine Azure SQL

▼

Technical Specifications

Core Engine: Built on Unity 2020.3 LTS with Microsoft Mixed Reality Toolkit (MRTK) for HoloLens 2 integration.
Computer Vision: Implemented Vuforia Engine using Cylindrical Image Targets to recognize and overlay expiration dates on physical tea bottles.
Data Pipeline: Developed a C# script to serialize user performance metrics (reaction time, error frequency) into JSON. Data is transmitted via UnityWebRequest to a PHP RESTful API and stored in Azure SQL for longitudinal analysis.
Scaffolding Algorithm: Designed a conditional logic system that automatically adjusts cue intensity (Visual Guide → Audio Hint → None) based on real-time success rates.

Research Outcomes (Published in ET&S)

Tau-U Analysis: Statistical analysis confirmed significant intervention effects for all three participants (Tau-U scores: 1.0, 0.88, 1.0).
Generalization: Participants successfully applied the "First-In, First-Out" logic to a novel task (arranging cookie boxes) without AR assistance.

System Design & Social Inclusion

MR Social Learning Support for Hearing Impaired

Visualized real-time audio cues and speech-to-text. Offloaded processing to Azure cloud to ensure low-latency performance on edge devices.

C# Azure Cognitive Services Client-Server Arch

▼

Project Overview

The Problem: Students with hearing impairments often struggle to identify sound direction and emotional tone in fast-paced inclusive classrooms, leading to social isolation.
The Solution: A Mixed Reality application that acts as a "Social Radar," visualizing invisible auditory information directly in the user's field of view.

Key Features

Real-Time Subtitles: Converts speech to text instantly using cloud-based recognition, allowing students to read what they cannot hear.
Sound Visualization: Uses directional arrows and color intensity to show where a sound is coming from and how loud it is (e.g., indicating an urgent or angry tone).
Award Winner: Recognized at the ASUS Think Next Award for its innovative application of technology to solve social equity issues.

Full Stack & AI Integration

Decision Support System for Major Selection

A web-based system guiding undergraduate major selection. Integrated GPT API to transform MBTI aptitude data into personalized career guidance.

PHP/MySQL OpenAI API JavaScript

▼

Project Overview

The Concept: Many students struggle to choose a major that aligns with their personality. This system bridges the gap between psychological aptitude tests and university department data.

Key Features

AI Analysis: Instead of simple keyword matching, the system uses Generative AI to analyze a student's Holland Code (RIASEC) scores and explain why specific departments are a good fit.
Data Integration: Combines employment statistics and curriculum data to provide a comprehensive view of potential career paths.
User Features: Includes a forum for peer discussion and a bookmarking system to save potential major choices.

Deep Learning

Nationwide Satellite Image Classification

Built a CNN pipeline to process nationwide satellite imagery. Automated land-use decision making by reducing manual review time with high-accuracy models.

Python ResNet-50 TensorFlow

▼

Project Overview

The Goal: Automating the classification of land use (e.g., distinguishing between forests, residential areas, and highways) to assist in urban planning and environmental monitoring.

Key Features

Deep Learning Model: Utilized the ResNet-50 architecture, a powerful Convolutional Neural Network, to identify complex patterns in satellite imagery.
High Accuracy: The model successfully learned to distinguish visually similar terrains, significantly reducing the time required for manual image review.
Automation: Created a pipeline that can process large batches of images, enabling efficient nationwide scale analysis.