INSTITUTE OF COMPUTER SCIENCE
UniTartuCS Student Projects
Back to projects

Self Service Checkout With Object Detection

In progress
machine-learningprototypeuser-experienceinteraction-design
Master·Project start: 11.02.2026·by: Hendrik Aruoja, Gregory Landrat, Buland Kumar Pradhan, Jens Jaagup Jäger, Joseph Jacob Kulathinal

Pages

Introduction

Introduction

SmartCheckout: Camera-Assisted Self-Checkout Prototype

University of Tartu · Digital Innovation Industry Project · MSc
Student Project Competition

Team: Gregory Landrat · Jens Jaagup Jäger · Joseph Jacob Kulathinal · Buland Kumar Pradhan · Hendrik Aruoja
Live demo: warm-development-space.lovable.app
Recommended demo device: iPad in landscape orientation
Project status: Functional prototype for research and usability testing

1. Executive Summary

SmartCheckout is a camera-assisted self-checkout prototype designed for convenience retail, especially small-basket environments such as fuel station shops. The project responds to a practical retail problem: self-checkout units are available, but customers often continue to choose staffed point-of-sale lanes even when their purchase should be quick.

The prototype explores a different interaction model. Instead of asking customers to find and scan every barcode one item at a time, SmartCheckout uses computer vision to propose a cart from a camera image. The customer then reviews the detected products, adjusts quantities if needed, and proceeds to payment.

The goal is not to remove staff from the store. The goal is to reduce avoidable friction at the self-checkout while keeping human help available when the system is uncertain or the customer needs support. For this reason, the prototype includes both an AI-assisted checkout flow and a classic barcode-style flow, allowing the team to compare speed, confidence, corrections, and completion rates.

2. Problem Context

Self-checkout should be a strong fit for convenience retail. Baskets are usually small, customers are often in a hurry, and store layouts are designed around fast visits. In practice, the expected speed advantage does not always translate into usage.

The issue is not only that self-checkout can be slow. The larger issue is that customers anticipate problems before they start. A customer who expects a failed scan, an unknown product lookup, an age-restricted item issue, a loyalty-card problem, or an embarrassing staff intervention may choose the staffed lane even when the queue is longer.

This makes self-checkout adoption a behavioral and operational problem, not just a hardware problem. If customers do not trust the process, the units remain underused. Low usage reduces the merchant's return on investment and makes further rollout harder to justify.

Project problem statement

Self-checkout transaction volume is too low compared with regular point-of-sale in convenience retail, limiting merchant return on investment and reducing willingness to invest in additional units.

3. Users and Stakeholders

Primary users

The main users are convenience-store customers choosing between self-checkout and staffed checkout during a purchase. Their decision is shaped by three questions:

  1. Will this be faster than the cashier?
  2. Will I be able to complete it without help?
  3. Will the machine make the process feel awkward, uncertain, or slow?

Secondary users

Store staff are also central to the experience. They are responsible for helping customers, resolving exceptions, approving restricted items, and keeping the checkout area moving. A better self-checkout flow should reduce unnecessary interventions without hiding situations where staff support is genuinely needed.

Business stakeholders

For retailers and technology providers, the relevant outcomes are higher self-checkout usage, shorter queues, fewer avoidable support calls, acceptable loss-prevention controls, and a solution that can scale without major new hardware investment.

4. Research Assumptions

The project started from several working assumptions about why customers avoid self-checkout in convenience retail. These assumptions need validation through observation and testing, but they shaped the prototype direction.

AssumptionWhy it mattersDesign response in SmartCheckout
Barcode scanning creates frictionCustomers must rotate products, find small codes, and rescan failed itemsCamera-first recognition proposes the cart from product appearance
First-time users fear getting stuckCustomers may avoid SCO because they do not know what will happen nextThree-step flow: scan, review, pay
Staff intervention feels like failureCalling staff can be embarrassing and slower than using the cashierOne-tap pager makes help explicit rather than disruptive
Product exceptions reduce trustAge-restricted items, prepared food, loyalty identification, and unknown products create uncertaintyReview step and exception-handling space for staff involvement
Different users need different levels of guidanceAge, language, and digital confidence affect adoptionLarge touch targets, plain language, and multilingual support
Retailers need measurable evidenceA convincing prototype should show whether the new flow performs betterAI and classic checkout modes are implemented side by side

5. Design Hypothesis

If self-checkout can move from item-by-item barcode hunting to camera-assisted cart confirmation, customers will be more willing to start and complete the self-checkout process.

The hypothesis depends on three conditions:

  1. Recognition must be fast enough to feel useful.
  2. The customer must remain in control through review and correction.
  3. Staff support must remain visible for edge cases.

SmartCheckout therefore treats AI as an assistant, not an invisible authority. The system proposes; the customer confirms.

6. Product Concept

SmartCheckout is built around a simple checkout loop:

Camera scan  ->  Cart proposal  ->  Customer review  ->  Payment  ->  Receipt

Step 1: Scan

The customer places products in view of the kiosk camera. The system attempts to identify multiple items from the frame and creates a proposed cart.

Step 2: Review

The customer checks the proposed cart, adjusts quantities, removes incorrect items, or requests staff assistance. This step is deliberately kept visible because trust depends on the customer seeing what the system believes it recognized.

Step 3: Pay

After confirming the cart, the customer proceeds to the payment screen and completes the transaction flow.

7. Prototype Features

AI-assisted checkout

The AI checkout path demonstrates how a camera-based flow could reduce barcode-scanning effort. It is intended for comparison against a classic scanning flow, not as a final production claim.

Route: /checkout/ai

Classic checkout comparison

The classic flow provides a baseline for measuring whether the AI-assisted flow is actually better. It supports the research goal by giving participants two comparable checkout experiences inside the same prototype.

Route: /checkout/classic

Timing analytics

The prototype records end-to-end checkout duration so the team can compare completion time between flows. Future testing should also track correction rate, staff-call rate, abandonment, and user preference.

Staff pager

A dedicated help action allows the customer to request support. This avoids hiding the need for staff and gives the prototype a more realistic retail operating model.

Product trainer

The admin-side trainer supports multi-shot product capture so the product catalog can improve over time. This is important for convenience retail, where packaging changes, local products, campaign items, and prepared-food options can make static product recognition brittle.

Accessibility-oriented kiosk UI

The interface uses large touch areas, clear visual hierarchy, and multilingual support. These choices are intended to reduce hesitation for first-time users and support customers with different levels of digital confidence.

8. Technical Architecture

SmartCheckout uses a hybrid recognition approach: local matching is preferred for known products, while a cloud AI model is used as a fallback for uncertain or long-tail cases.

Camera input
    |
    v
Image preprocessing
    |
    v
Local recognizer
MobileNet embeddings + pgvector catalog search
    |
    |-- confident match --> Cart proposal
    |
    |-- uncertain match --> Gemini 2.5 Flash fallback
                              |
                              v
                         Cart proposal
                              |
                              v
Customer review and correction
                              |
                              v
Payment flow + timing analytics

Technology stack

LayerImplementation
FrontendReact 18, Vite, Tailwind CSS, shadcn/ui
AI vision fallbackGoogle Gemini 2.5 Flash
Local recognitionMobileNet embeddings with pgvector similarity search
BackendLovable Cloud / Supabase for realtime data, auth, and edge functions
Staff supportRealtime pager notifications
Admin toolsProduct trainer with multi-shot capture
Research instrumentationCheckout-flow timing analytics

9. Why This Approach Is Different

Most self-checkout improvements make the existing barcode process slightly better. SmartCheckout tests whether the interaction model itself should change.

The key difference is that the system starts from a visual cart proposal rather than from manual scanning. This matters because the most frustrating parts of self-checkout often happen before payment: finding barcodes, rescanning items, searching produce lists, handling unexpected bagging-area prompts, and waiting for staff after an error.

SmartCheckout does not assume that AI should fully automate checkout. Instead, it uses AI to reduce repetitive scanning work while keeping the final cart confirmation with the customer. This keeps the experience transparent and gives the retailer a clear point for validation, correction, and staff intervention.

10. Demo

Open the live prototype:

https://warm-development-space.lovable.app/

Recommended device: iPad in landscape orientation.

Demo routes:

  • AI-assisted checkout: /checkout/ai
  • Classic checkout: /checkout/classic

References