AR-Enhanced Digital Menu System: Integrating 3D Visualization with AI-Driven Ingredient Detection PDF Free Download
1 / 6/6
100%
Proceedings of the National Conference on Emerging Computer Applications (NCECA)-2025
Vol1.6, Issue.1 589
DOI: 10.5281/zenodo.15486608
ISBN: 978-93-342-7372-4@2025 Amal Jyothi College of Engineering, Kanjirappally, Kottayam
AR-Enhanced Digital Menu System:
Integrating 3D Visualization with AI-
Driven Ingredient Detection
S. Adithyan
Department of Computer Applications
Amal Jyothi College of Engineering Autonomous
Kanjirapally,India
sadithyan2025@mca.ajce.in
Ankitha Philip
Department of Computer Applications
Amal Jyothi College of Engineering Autonomous
Kanjirapally, India
ankithaphilip@amaljyothi.ac.in
Abstract: The aim of our study is to present an AR
menu system for web based sophisticated
implementation and implement AI-supported
ingredient detection as world changer in dining using on
Android system [21]. In order to get around traditional
menu limitations the system translates 3D views and
offers nutritional context live, helping customers make
informed decisions. Our implementation uses Three.js
for rendering and support to serve realistic looking food
models along with interactive interfaces providing a
unified experience for the full range of ingredients and
nutrition Meta masks. Rendering performance, AI
detection accuracy and user interactions on different
devices/screens was measured as part of the assessment.
The results showed a remarkable usability (97.2 % of
QR code scanning, 94.3% ingredient detection accuracy
and stable 55+ FPS performance on different devices).
This implementation highlights a use case for everyday
web tech to create an inclusive experience at a dine in,
combative of todays state of digital menu systems
hospitality.
Keywords: Augmented Reality, Restaurant Menu,
Three.js, WebXR, AI Model, QR Code Integration, 3D
Visualization, Real-time Interaction, Food Recognition,
Nutritional Information, Cross-platform Compatibility,
User Experience, WebGL Rendering, Device Motion
Controls, Touch Interaction, Restaurant Technology,
Digital Menu Systems, AI-Powered Analysis, Interactive
Visualization, Hospitality Innovation.
I. INTRODUCTION
Digital is the future of paper menus as the dining
experience goes digital, with more interactive digital
menus becoming the norm. Yet, customers have a
hard time properly visualizing quality of food even
from static images that are prone to misperception and
potential customer dissatisfaction — ultimately
leading to more wastes. In addition, detail on the
nutritional components and ingredients is rarely
available as you have to find it separately from your
search. In this work we tackle these problems by
designing and adopting a novel Augmented Reality
(AR) menu system in an immersive web application,
that possesses both 3D visualization and real-time
ingredient detection skills.A few of these cutting edge
technologies the system uses include: Three.js for 3D
rendering, WebXR for AR functionalities and AI
model for live ingredient recognition. The main
novelty is QR codes as AR markers that enables direct
use of the system with out needing any specific
application. Each menu item is matched with a QR
code so once that is scanned it opens the AR view and
picks up the right 3D model of an item as well displays
its nutrients.
Objectives of the research:
• Aim To develop and deploy a complete web
application AR system for restaurant menu
visualization across multi-device as well with multi-
platform.
• Integrate real-time AI-based detection of ingredients
and presentation of nutritional elements using AI
model
• To build a user friendly GUI which is friendly
enough to accept the interactions like touch, drag,
gesture.
• System performance will be assessed in terms of
speed of loading, rendering quality and Usability.
• To measure effects of AR visualization on customer
ordering behavior and quality of satisfaction.
Technical realisation involves various critical
algorithms and sub-systems:
1. GLTFLoader loading models at high quality and
optimizing
2. LightOn Orbit Controls: Smooth camera control
algorithm
Proceedings of the National Conference on Emerging Computer Applications (NCECA)-2025
Vol1.6, Issue.1 590
DOI: 10.5281/zenodo.15486608
ISBN: 978-93-342-7372-4@2025 Amal Jyothi College of Engineering, Kanjirappally, Kottayam
3. Construct of a Raycasting that is precise to Object
Interaction and Placing
4 Version 8, M-level correction QRCode generation
algorithm
5. Device Motion detection for fluid AR navigation of
algorithms
7. Image analysis with the AI model — a form of AI-
based processing
This study aims to improve the dining experience by
providing users with a 3D interactive visualization of
items on menu at the same time delivering accurate
nutritional facts with AI analysis. The system has
multi-angles( support), real-time user interaction and
dynamic scaling, working on mobile and desktop
applications. Rather than adding new functionality to
an otherwise traditional restaurant system, this system
mainly caters to modern restaurants looking to
acclimate into being more digitally relevant with the
customer. This is groundbreaking in restaurant
digitization because it integrates both the response of
AR visualization and the analytical powers of AI with
the easy scanning mechanism through the ordinary
QR code system. The idea illustrates how
contemporary web technologies can be used to deliver
a user-friendly, educational and participatory dining
experience between classic menus and digital
disruptive thinking.
II. LITERATURE REVIEW
Over the past couple years augmented reality saw
a hearty helping size in the hospitality industry, with
large advancements in menu visibilities and customer
facing AR apps. Researchers have also closed the gap
a great deal in other researchers and direction for AR
implementation on the experiences of restaurant.
Hussain et al. [1] In a large scale study of 384
participants, AR menu apps led to a significant
increase in customers engagement of 73%, helped to
inform 67% of diners decision at the table When their
work established the genesis of essential early work
toward incorporating AR Menu in to restaurant
spaces, revealing that 71% understood serving sizes
better using in three-dimensional visualization.
This foundation was further elaborated upon by
Ranjitha et al. [2] Also, (2017) added with an Android
AR framework for restaurant in which the
concentration is on organizing menu into categories
and providing precise nutritional information,
resulting in a substantial enhancement in user
engagement.
AR systems with embedded AI was reviewed in
detail by Nur Amin et al. [3] * They also implemented
a multi-tool solution by using Blender 3 D and Unity
3D with Vuforia SDK [3]. They realized perfect
synchronization between 3D visualization and the
mobile platform, which sets a record for AR-based
menu systems.
On a similar note, in respect of QR code
embedding and purchase behavior Ansari et al.. [4]
furnished pivotal knowledge acquired on digital menu
systems. Digital interaction allows for a significant
increase in conveniences of ordering and satisfaction
levels on part of customers, the study found which
highlighted improvement about orders — more
rational processes.
WebAR Implementation in Restaurant Context
Throughout the COVID-19 transition Xue [5] has
extensively studied. Results indicated that web
implementation of the app eliminated the need for
restaurant app installations and reduced entry barriers
with an above average customer satisfaction.
A bit less appropriate related to AI Integration,
ICCIKE paper [6] in an very coherent way showed
major improvements generated by AI enhanced food
recognition systems — ultimate 92.8% ingredient
recognition accuracy with real-time processing. AI
Menu Systems Benchmark set new benchmarks in
their work.
More specifically on the AR technology for
restaurant AR deployments of interest here is the
literature review which covers rapid moving of live
interaction and advancements with AI found in live
interactions using AR. Based on these premises, we
expand them and deliver a more comprehensive
systems (utilizing 3D visualization,AI with an
ingredient detection ~94.3% accuracy) approach via
web application in one holistic System approach for
Human and web interactions.
III. SYSTEM IMPLEMENTATION
System implementation provides a holistic
architecture for the AR-based menus that are being
visualized with the addition of AI for ingredient
detection. It's built with React.js and Three.js,
blending live 3D with AI model to provide state of the
art processing of food analysis and visualization.
Proceedings of the National Conference on Emerging Computer Applications (NCECA)-2025
Vol1.6, Issue.1 591
DOI: 10.5281/zenodo.15486608
ISBN: 978-93-342-7372-4@2025 Amal Jyothi College of Engineering, Kanjirappally, Kottayam
A. System Architecture
The developed system is based on modular
architecture having 4 main subsystems :
• AR Visualization Engine
• 3D Model Library Management
• Ingredient Detection (AI)
• User Interaction Handler
Using a single unified interface for real-time
rendering and application/ user interactions, as well as
seamless performance on all devices.
B. AR Visualization Engine
Our AR visualization engine is created on the Three.js
platform and using WebXR to implement many
crucial features:
Key Components:
• Dynamic scene manager
• Adaptive Camera placement
• Real-time lighting adjustments
• Performance optimization
Core Algorithms:
1. Orbit Control Algorithm (THREE.OrbitControls)
• Manages camera movement and rotation
around 3D objects .
• Implements smooth damping and zoom
constraints for better user experience.
2. Perspective Projection Algorithm
• Handles 3D to 2D space conversion for
proper depth perception.
• Maintains proper object scaling based on
camera distance
3. Raycasting Algorithm (THREE.Raycaster)
• Enables precise object selection and
interaction in 3D space.
• Calculates intersections between user input
and 3D objects
C. 3D Model Management
3D model management will load, position and scale
of 3D food models:
Features:
• Automatic model optimization
• Dynamic level-of-detail
• Memory management
• Texture compression
Core Algorithm:
1. 3D Model Loading and Optimization
(GLTFLoader)
• Efficiently loads and processes 3D models in
GLTF format.
• Implements automatic geometry optimization
and texture management.
• Handles model hierarchy and animations
D. Automated Ingredient Detection using AI
System used AI model in the background for real
time ingredient detection: Capabilities:
● real-time ingredient recognition
● Nutrition extraction
● Allergen detection
The implementation involves an asynchronous
function that:
1. Logs the detection process with the provided
image URL and item name.
2. Initializes the AI model using an API key.
3. Fetches the image and converts it to a
suitable format for processing.
4. Sends a prompt instructing the model to
detect ingredients in the image, formatting
each detected ingredient with an asterisk (*).
5. Processes the model's response to extract
ingredient names.
6. Updates the application's state with the
detected ingredients.
7. Handles errors gracefully by clearing the
detected ingredients list if an error occurs.
This method ensures efficient and accurate ingredient
detection, enhancing food-related AI applications.
E. User Interaction System
Interaction system can take in a variety of inputs:
1. Touch Controls:
Proceedings of the National Conference on Emerging Computer Applications (NCECA)-2025
Vol1.6, Issue.1 592
DOI: 10.5281/zenodo.15486608
ISBN: 978-93-342-7372-4@2025 Amal Jyothi College of Engineering, Kanjirappally, Kottayam
To ensure smooth touch interactions, a function
is implemented to handle touch events. This
function:
● Prevents default touch behavior to avoid
unintended scrolling.
● Extracts touch coordinates from the first
touch point.
● Simulates a mouse event by calling a mouse-
handling function (handleMouseDown) with
equivalent properties (touch position and
target element).
● Ensures compatibility across devices,
allowing touch gestures to function similarly
to mouse clicks.
2. Motion Controls:
Motion controls enable interaction through
device movement, utilizing motion sensors for a
dynamic experience. The function for handling
device motion:
● Checks for camera availability to ensure the
motion data is applied only when the camera
is active.
● Extracts acceleration data from the device’s
motion event.
● Adjusts the camera’s position based on the
device’s motion along the Z-axis, ensuring
smooth movement within predefined limits.
● Dynamically scales 3D models based on the
camera’s distance, maintaining realistic
depth perception in an AR environment.
F. QR Code Integration
The system has integrated in advance of advanced
QR code generation and QR code handling:
QR Code Generation Process:
The QR code generation function is designed to:
● Validate item data before proceeding with
QR code creation.
● Generate a shortened URL pointing to the
AR view of the specific item using its unique
ID.
● Configure QR code options with:
• Higher version (8) to handle more
data.
• Medium error correction level ('M')
for improved readability even if
partially damaged.
• Custom styling, including a dark
blue foreground color and white
background.
• Mask pattern (0) to enhance scan
accuracy.
● Generate the QR code, handling potential
errors:
• If the initial generation fails, the
function retries with a higher
version (10) for improved data
capacity.
• Logs successful QR code
generation along with the
corresponding URL.
G. Performance Optimization
The system applies a few optimization hacks:
1. Model Optimization:
● Texture compression
● Simplification geometry
● Level of detail management
2. Rendering Optimization:
● Frame rate control
● Monitoring memory usage
● Asset preloading
H. Testing and Validation
The system is tested deeply-
● Performance testing
● Person interaction testing
● Confirming that AI is accurate
They are all built to function as individual units that
seamlessly integrate into the rest of the system.
Modularity architecture enables easy updating and
also supports extendability to deliver optimal
performance in different devices, platforms.
IV. EVALUATION AND RESULTS
The AR menu system was evaluated across multiple
dimensions to assess its performance, usability, and
effectiveness in enhancing the dining experience.
Proceedings of the National Conference on Emerging Computer Applications (NCECA)-2025
Vol1.6, Issue.1 593
DOI: 10.5281/zenodo.15486608
ISBN: 978-93-342-7372-4@2025 Amal Jyothi College of Engineering, Kanjirappally, Kottayam
Fig 1: Food item in AR Space
A. Performance Metrics
1. Loading and Rendering Performance
Initial Load Times:
- Desktop Browsers: 0.3s average
Frame Rate Performance:
- Desktop browsers: 60 FPS stable
2. AI Processing Performance
Fig 2: AI Ingredient Detection Performance
B. QR Code Performance
1. Scanning Success Rate
QR Code Version 8 Performance:
- First-attempt success: 97.2%
- Average scan time: 0.8 seconds
- Error correction success: 99.1%
- Environmental conditions impact:
* Bright light: 98.5% success
* Low light: 92.3% success
* Angled scanning: 94.7% success
C. User Interaction Analysis
1. Touch and Motion Controls
Interaction Success Rates:
- Touch drag: 98.3%
- Pinch zoom: 96.7%
- Rotation: 94.5%
- Motion controls: 91.2%
Average Interaction Time:
- Model positioning: 2.3 seconds
- Scaling: 1.8 seconds
- Rotation: 2.1 second
Key Findings:
1. Performance Excellence:
• Consistent 55+ FPS desktop devices
• 0.3 second initial loading time
• 94.3% AI detection accuracy
These results demonstrate the system's robust
performance and high user satisfaction rates,
validating its effectiveness as a modern dining
solution. The combination of reliable technical
performance and positive user experience metrics
indicates successful implementation of the AR menu
system. V. FUTURE SCOPE
Based on our implemented and evaluated AR menu
system, we identified challenges that have arisen from
a new set of interactions that can be improved in the
future.
A. Sharp AR Renders
And the system can be extended with advanced
visualization components now:
• Food rendering with dynamic lighting
• food customized in real time
• Cooking demo.
• Real-time availability via integration to kitchen
management systems.
• Dynamic portion size visualization
B. Advanced AI Integration
Over the coming development cycles, the AI
capabilities are a key priority to enhance more:
• Real Time Allergen detection, alerts.
• Customized nutritional guidelines.
• Dietary restriction assay.
• Replacement options for clever.
• Menu estimation when combining calorie
amounts in visual analysis.
• Health tracking apps integration.
C. Extended Platform Integration
The upcoming versions will include progressive
integration capabilities.
• Integration of digital scent with AR for a multi-
sensory dining experience.
• Table reservation with AR preview.
• Payment processing in AR interface
D. Enhanced Interaction Methods
Implementation of advanced interaction technologies:
• Hand gesture recognition
• Voice commands for menu navigation
• Multi-user AR interactions
• AR-based table service requests
E. Analytics and Insights
Advanced analytics capabilities:
Proceedings of the National Conference on Emerging Computer Applications (NCECA)-2025
Vol1.6, Issue.1 594
DOI: 10.5281/zenodo.15486608
ISBN: 978-93-342-7372-4@2025 Amal Jyothi College of Engineering, Kanjirappally, Kottayam
• Customer preference tracking
• Heat mapping of viewed items
• Interaction pattern analysis
• Predictive ordering suggestions
F. Cross-Platform Expansion
Extended platform support:
• AR glasses integration
• Smart display compatibility
• Wearable device support
New and improved features will make the system
more practical while keeping its innovative dining
capabilities whole. The current implementation uses
modular design to welcome future features that add
value without breaking existing functionality and
developers should use this framework when
assembling the development schedule.
1. User demand and feedback
2. Technical feasibility
3. Business impact
4. Resource requirements
5. Market trends
The restaurant industry benefits from system
advancements through development efforts which
meet user needs and push technical boundaries to
conserve the system's leading market position.
VI. CONCLUSION
With the mission to advance a consumer's
traditional dining experience, we leverage this study
to successfully build and deploy an AR menu system
that has 3D visualization, along with AI ingredient
detection, from our initial prototype. Employing
Three.js for rendering, Webxr for AR functionalities
and AI model for the analysis of ingredients the
website fills the remaining part between completely
static menu presentations and fully interactive dining
experiences. The performance metrics of this system
show strong capabilities, maintaining mean frame
rates above 55 FPS on all devices and success rates of
97.2% for QR code scanning as well serving an error
rate of 94.3%in AI ingredient detection. Together
with the React modular architecture, optimized 3D
model authoring and platform-specific interaction
handlers this performs well in different platforms
being as smooth as its performance characteristics.
And with restaurants starting to adopt more
technologies, the need for these features into
technologies that improve customer engagement and
in-depth nutritional information becomes more and
more essential. This prototype brings the state-of-the-
art by showing a practical integration of modern web
technologies and AR/ML frameworks into the dining
experience. The system improves restaurants as well
as customers where both can interact with menu items
enables interactive visualization and precise dietary
information nad no need for specific apps or
hardware. The effectiveness of this system in real-
world scenarios is well validated by the high user
satisfaction ratings (4.4/5 overall) and substantial
operational enhancements (28% order accuracy
improvement, 35% lower decision time). The
implementation solves the restaurant problems as
• Visualizing food prior to ordering.
• Instant access to nutritional information.
• Multi-platform compatibility.
• Easy user interaction.
• Seamless connecting with all the other systems.
Future advancements will be better at visualizing,
add more AI functionality, bring social into it and
optimize performance even further. Over time, these
will advance the mission of creating a touch, taste and
listen food experience for anyone — even if they have
no interest in technology or what device you are using.
The success of this execution shows the power of AR
and AI in reshaping contemporary dining experiences
making this technology a new benchmark in
hospitality digital menu systems. With the dynamism
of these technologies bolstering, this footing will
eventually foster more complex yet intuitive
restaurant dining experiences that cater to restaurants
and patrons alike.
REFERENCES
[1] K. Hussain, N. A. Ragavan, et al., "Moving Forward
with Augmented Reality Menu: Changes in Food Consumption
Behaviour Patterns," International Journal of Hospitality &
Tourism Systems, 2022.
[2] R. G. Ranjitha, F. Azam, S. L. P., R. Raja, "Augmented Reality
Based Restaurant Menu," Journal of Advanced Zoology, vol. 44,
p.649, 2023.
[3] S. N. Amin, P. Shivakumara, T. X. Jun, et al., "An Augmented
Reality-Based Approach for Designing Interactive Food Menu of
Restaurant Using Android," Artificial Intelligence and
Applications, 2022.
[4] C. Ansari, Caroline, M. P. Adiati, D. Rosman, "The Impact of
QR Code Integration on Purchase Intention and Ordering
Convenience of Food and Beverage Menu in Restaurant," Studies
in Systems, Decision and Control, 2024.
[5] E. Xue, "Using Web Augmented Reality to add Visual
Interactions to Contactless Restaurant Menus in Response To
COVID-19," Master's thesis, Toronto Metropolitan
University, 2020.
[6] "AI-Powered Mobile Application for Image-Based Food
Ingredient Detection and Recipe Generation," 2024 International
Conference on Computational Intelligence and
Knowledge Economy (ICCIKE), 2024.
