ML Kit Vision APIs in Flutter: Face Detection & OCR

Introduction to ML Kit Vision

Google ML Kit brings Google's machine learning expertise to mobile developers in a powerful and easy-to-use package. The Vision APIs provide ready-to-use models for common computer vision tasks, running either on-device for real-time performance or in the cloud for enhanced accuracy. With Flutter's cross-platform capabilities, you can implement sophisticated vision features that work seamlessly on both iOS and Android.

ML Kit's Vision APIs excel in scenarios requiring real-time processing, privacy-sensitive applications, and offline functionality. Whether you're building a document scanner, a face filter app, or an accessibility tool, ML Kit provides the foundation for powerful computer vision features without the complexity of training your own models.

Setting Up ML Kit in Flutter

Let's start by setting up ML Kit in your Flutter project. We'll use the official google_ml_kit package which provides a unified interface for all ML Kit APIs.

1. Add Dependencies

# pubspec.yaml
dependencies:
  flutter:
    sdk: flutter
  google_ml_kit: ^0.16.0
  camera: ^0.10.5
  image_picker: ^1.0.0
  path_provider: ^2.1.0
  permission_handler: ^11.0.0

2. Platform Configuration

Android Setup

// android/app/build.gradle
android {
    compileSdkVersion 33
    
    defaultConfig {
        minSdkVersion 21  // ML Kit requires minimum SDK 21
        targetSdkVersion 33
    }
}

iOS Setup

# ios/Podfile
platform :ios, '12.0'  # ML Kit requires iOS 10.0+

# Add camera usage description to Info.plist
# NSCameraUsageDescription: "This app needs camera access for ML Kit features"

3. Permissions Management

import 'package:permission_handler/permission_handler.dart';

class PermissionManager {
  static Future requestCameraPermission() async {
    final status = await Permission.camera.request();
    return status == PermissionStatus.granted;
  }
  
  static Future requestStoragePermission() async {
    final status = await Permission.storage.request();
    return status == PermissionStatus.granted;
  }
  
  static Future checkAndRequestPermissions() async {
    final cameraGranted = await requestCameraPermission();
    final storageGranted = await requestStoragePermission();
    
    return cameraGranted && storageGranted;
  }
}

Face Detection Implementation

Face detection is one of the most popular ML Kit features, enabling you to detect faces and facial landmarks in images and video streams. Let's implement a comprehensive face detection solution.

Original Image

Face Detected

Face Detector Service

import 'package:google_ml_kit/google_ml_kit.dart';
import 'package:camera/camera.dart';
import 'dart:ui' as ui;
import 'dart:io';

class FaceDetectorService {
  late FaceDetector _faceDetector;
  bool _isDetecting = false;
  
  FaceDetectorService() {
    _initializeDetector();
  }
  
  void _initializeDetector() {
    _faceDetector = GoogleMlKit.vision.faceDetector(
      FaceDetectorOptions(
        enableContours: true,
        enableClassification: true,
        enableLandmarks: true,
        enableTracking: true,
        minFaceSize: 0.1,
        performanceMode: FaceDetectorMode.accurate,
      ),
    );
  }
  
  Future> detectFacesFromImage(File imageFile) async {
    final inputImage = InputImage.fromFile(imageFile);
    return await _processFaceDetection(inputImage);
  }
  
  Future> detectFacesFromCamera(CameraImage image, CameraDescription camera) async {
    if (_isDetecting) return [];
    _isDetecting = true;
    
    try {
      final WriteBuffer allBytes = WriteBuffer();
      for (final Plane plane in image.planes) {
        allBytes.putUint8List(plane.bytes);
      }
      final bytes = allBytes.done().buffer.asUint8List();
      
      final Size imageSize = Size(image.width.toDouble(), image.height.toDouble());
      
      final InputImageRotation imageRotation = 
          _rotationIntToImageRotation(camera.sensorOrientation);
      
      final inputImageFormat = 
          InputImageFormatValue.fromRawValue(image.format.raw) ?? InputImageFormat.nv21;
      
      final planeData = image.planes.map(
        (Plane plane) {
          return InputImagePlaneMetadata(
            bytesPerRow: plane.bytesPerRow,
            height: plane.height,
            width: plane.width,
          );
        },
      ).toList();
      
      final inputImageData = InputImageData(
        size: imageSize,
        imageRotation: imageRotation,
        inputImageFormat: inputImageFormat,
        planeData: planeData,
      );
      
      final inputImage = InputImage.fromBytes(
        bytes: bytes,
        inputImageData: inputImageData,
      );
      
      return await _processFaceDetection(inputImage);
    } finally {
      _isDetecting = false;
    }
  }
  
  Future> _processFaceDetection(InputImage inputImage) async {
    try {
      final faces = await _faceDetector.processImage(inputImage);
      return faces;
    } catch (e) {
      print('Error detecting faces: $e');
      return [];
    }
  }
  
  InputImageRotation _rotationIntToImageRotation(int rotation) {
    switch (rotation) {
      case 90:
        return InputImageRotation.rotation90deg;
      case 180:
        return InputImageRotation.rotation180deg;
      case 270:
        return InputImageRotation.rotation270deg;
      default:
        return InputImageRotation.rotation0deg;
    }
  }
  
  void dispose() {
    _faceDetector.close();
  }
}

// Face information model
class FaceInfo {
  final Rect boundingBox;
  final double? smilingProbability;
  final double? leftEyeOpenProbability;
  final double? rightEyeOpenProbability;
  final double? headEulerAngleX;
  final double? headEulerAngleY;
  final double? headEulerAngleZ;
  final Map landmarks;
  final Map contours;
  
  FaceInfo({
    required this.boundingBox,
    this.smilingProbability,
    this.leftEyeOpenProbability,
    this.rightEyeOpenProbability,
    this.headEulerAngleX,
    this.headEulerAngleY,
    this.headEulerAngleZ,
    required this.landmarks,
    required this.contours,
  });
  
  factory FaceInfo.fromFace(Face face) {
    return FaceInfo(
      boundingBox: face.boundingBox,
      smilingProbability: face.smilingProbability,
      leftEyeOpenProbability: face.leftEyeOpenProbability,
      rightEyeOpenProbability: face.rightEyeOpenProbability,
      headEulerAngleX: face.headEulerAngleX,
      headEulerAngleY: face.headEulerAngleY,
      headEulerAngleZ: face.headEulerAngleZ,
      landmarks: face.landmarks,
      contours: face.contours,
    );
  }
  
  bool get isSmiling => (smilingProbability ?? 0) > 0.8;
  bool get bothEyesOpen => 
      (leftEyeOpenProbability ?? 0) > 0.8 && 
      (rightEyeOpenProbability ?? 0) > 0.8;
}

Face Detection UI

import 'package:flutter/material.dart';
import 'package:camera/camera.dart';

class FaceDetectionScreen extends StatefulWidget {
  @override
  _FaceDetectionScreenState createState() => _FaceDetectionScreenState();
}

class _FaceDetectionScreenState extends State {
  final FaceDetectorService _faceDetectorService = FaceDetectorService();
  CameraController? _cameraController;
  List _faces = [];
  bool _isDetecting = false;
  
  @override
  void initState() {
    super.initState();
    _initializeCamera();
  }
  
  Future _initializeCamera() async {
    final cameras = await availableCameras();
    final frontCamera = cameras.firstWhere(
      (camera) => camera.lensDirection == CameraLensDirection.front,
      orElse: () => cameras.first,
    );
    
    _cameraController = CameraController(
      frontCamera,
      ResolutionPreset.high,
      enableAudio: false,
    );
    
    await _cameraController!.initialize();
    
    _cameraController!.startImageStream((CameraImage image) async {
      if (!_isDetecting) {
        _isDetecting = true;
        
        final faces = await _faceDetectorService.detectFacesFromCamera(
          image,
          frontCamera,
        );
        
        setState(() {
          _faces = faces;
        });
        
        _isDetecting = false;
      }
    });
    
    setState(() {});
  }
  
  @override
  Widget build(BuildContext context) {
    if (_cameraController == null || !_cameraController!.value.isInitialized) {
      return Scaffold(
        body: Center(child: CircularProgressIndicator()),
      );
    }
    
    return Scaffold(
      appBar: AppBar(
        title: Text('Face Detection'),
        actions: [
          IconButton(
            icon: Icon(Icons.switch_camera),
            onPressed: _switchCamera,
          ),
        ],
      ),
      body: Stack(
        fit: StackFit.expand,
        children: [
          CameraPreview(_cameraController!),
          CustomPaint(
            painter: FacePainter(
              faces: _faces,
              imageSize: _cameraController!.value.previewSize!,
            ),
          ),
          Positioned(
            bottom: 20,
            left: 20,
            right: 20,
            child: _buildInfoCard(),
          ),
        ],
      ),
    );
  }
  
  Widget _buildInfoCard() {
    if (_faces.isEmpty) {
      return Card(
        child: Padding(
          padding: EdgeInsets.all(16),
          child: Text(
            'No faces detected',
            textAlign: TextAlign.center,
          ),
        ),
      );
    }
    
    final face = _faces.first;
    final faceInfo = FaceInfo.fromFace(face);
    
    return Card(
      child: Padding(
        padding: EdgeInsets.all(16),
        child: Column(
          mainAxisSize: MainAxisSize.min,
          children: [
            Text(
              'Face Detected',
              style: TextStyle(fontWeight: FontWeight.bold),
            ),
            SizedBox(height: 8),
            Row(
              mainAxisAlignment: MainAxisAlignment.spaceAround,
              children: [
                _buildFeatureChip(
                  'Smiling',
                  faceInfo.isSmiling,
                  Icons.sentiment_satisfied,
                ),
                _buildFeatureChip(
                  'Eyes Open',
                  faceInfo.bothEyesOpen,
                  Icons.visibility,
                ),
              ],
            ),
            if (face.headEulerAngleY != null)
              Text(
                'Head rotation: ${face.headEulerAngleY!.toStringAsFixed(1)}°',
                style: TextStyle(fontSize: 12),
              ),
          ],
        ),
      ),
    );
  }
  
  Widget _buildFeatureChip(String label, bool active, IconData icon) {
    return Chip(
      avatar: Icon(
        icon,
        size: 18,
        color: active ? Colors.green : Colors.grey,
      ),
      label: Text(label),
      backgroundColor: active ? Colors.green.shade50 : Colors.grey.shade200,
    );
  }
  
  Future _switchCamera() async {
    // Implementation for switching between front and back camera
  }
  
  @override
  void dispose() {
    _cameraController?.dispose();
    _faceDetectorService.dispose();
    super.dispose();
  }
}

// Custom painter for drawing face boundaries
class FacePainter extends CustomPainter {
  final List faces;
  final Size imageSize;
  
  FacePainter({required this.faces, required this.imageSize});
  
  @override
  void paint(Canvas canvas, Size size) {
    final Paint paint = Paint()
      ..style = PaintingStyle.stroke
      ..strokeWidth = 3.0
      ..color = Colors.green;
    
    final Paint landmarkPaint = Paint()
      ..style = PaintingStyle.fill
      ..strokeWidth = 2.0
      ..color = Colors.red;
    
    for (final Face face in faces) {
      // Scale face bounding box to screen size
      final rect = _scaleRect(
        rect: face.boundingBox,
        imageSize: imageSize,
        widgetSize: size,
      );
      
      // Draw bounding box
      canvas.drawRect(rect, paint);
      
      // Draw facial landmarks
      void paintLandmark(FaceLandmarkType type) {
        final landmark = face.landmarks[type];
        if (landmark != null) {
          canvas.drawCircle(
            Offset(
              landmark.position.x * size.width / imageSize.width,
              landmark.position.y * size.height / imageSize.height,
            ),
            5,
            landmarkPaint,
          );
        }
      }
      
      paintLandmark(FaceLandmarkType.leftEye);
      paintLandmark(FaceLandmarkType.rightEye);
      paintLandmark(FaceLandmarkType.noseBase);
      paintLandmark(FaceLandmarkType.leftMouth);
      paintLandmark(FaceLandmarkType.rightMouth);
      paintLandmark(FaceLandmarkType.bottomMouth);
      
      // Draw face contours if available
      if (face.contours.isNotEmpty) {
        final contourPaint = Paint()
          ..style = PaintingStyle.stroke
          ..strokeWidth = 1.0
          ..color = Colors.blue;
        
        face.contours.forEach((type, contour) {
          if (contour != null && contour.points.isNotEmpty) {
            final path = Path();
            final firstPoint = contour.points.first;
            path.moveTo(
              firstPoint.x * size.width / imageSize.width,
              firstPoint.y * size.height / imageSize.height,
            );
            
            for (final point in contour.points.skip(1)) {
              path.lineTo(
                point.x * size.width / imageSize.width,
                point.y * size.height / imageSize.height,
              );
            }
            
            canvas.drawPath(path, contourPaint);
          }
        });
      }
    }
  }
  
  Rect _scaleRect({
    required Rect rect,
    required Size imageSize,
    required Size widgetSize,
  }) {
    final double scaleX = widgetSize.width / imageSize.width;
    final double scaleY = widgetSize.height / imageSize.height;
    
    return Rect.fromLTRB(
      rect.left * scaleX,
      rect.top * scaleY,
      rect.right * scaleX,
      rect.bottom * scaleY,
    );
  }
  
  @override
  bool shouldRepaint(FacePainter oldDelegate) {
    return oldDelegate.faces != faces;
  }
}

Text Recognition (OCR)

ML Kit's text recognition API can recognize text in any Latin-based language and more. It's perfect for scanning documents, extracting information from images, and building accessibility features.

Text Recognition Service

class TextRecognitionService {
  late TextRecognizer _textRecognizer;
  
  TextRecognitionService({bool useCloudModel = false}) {
    _textRecognizer = GoogleMlKit.vision.textRecognizer(
      script: useCloudModel ? TextRecognitionScript.chinese : TextRecognitionScript.latin,
    );
  }
  
  Future recognizeText(File imageFile) async {
    try {
      final inputImage = InputImage.fromFile(imageFile);
      final recognizedText = await _textRecognizer.processImage(inputImage);
      return recognizedText;
    } catch (e) {
      print('Error recognizing text: $e');
      return null;
    }
  }
  
  Future recognizeTextFromBytes(
    Uint8List bytes,
    InputImageData imageData,
  ) async {
    try {
      final inputImage = InputImage.fromBytes(
        bytes: bytes,
        inputImageData: imageData,
      );
      return await _textRecognizer.processImage(inputImage);
    } catch (e) {
      print('Error recognizing text from bytes: $e');
      return null;
    }
  }
  
  List extractTextBlocks(RecognizedText recognizedText) {
    return recognizedText.blocks.map((block) {
      return TextBlock(
        text: block.text,
        boundingBox: block.boundingBox,
        recognizedLanguages: block.recognizedLanguages,
        lines: block.lines.map((line) {
          return TextLine(
            text: line.text,
            boundingBox: line.boundingBox,
            elements: line.elements.map((element) {
              return TextElement(
                text: element.text,
                boundingBox: element.boundingBox,
              );
            }).toList(),
          );
        }).toList(),
      );
    }).toList();
  }
  
  String extractPhoneNumbers(String text) {
    final phoneRegex = RegExp(
      r'(\+?\d{1,3}[-.\s]?)?\(?\d{1,4}\)?[-.\s]?\d{1,4}[-.\s]?\d{1,9}',
    );
    final matches = phoneRegex.allMatches(text);
    return matches.map((match) => match.group(0)!).join(', ');
  }
  
  String extractEmails(String text) {
    final emailRegex = RegExp(
      r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',
    );
    final matches = emailRegex.allMatches(text);
    return matches.map((match) => match.group(0)!).join(', ');
  }
  
  String extractUrls(String text) {
    final urlRegex = RegExp(
      r'https?://(?:www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b(?:[-a-zA-Z0-9()@:%_\+.~#?&\/=]*)',
    );
    final matches = urlRegex.allMatches(text);
    return matches.map((match) => match.group(0)!).join(', ');
  }
  
  void dispose() {
    _textRecognizer.close();
  }
}

// Models for structured text data
class TextBlock {
  final String text;
  final Rect boundingBox;
  final List recognizedLanguages;
  final List lines;
  
  TextBlock({
    required this.text,
    required this.boundingBox,
    required this.recognizedLanguages,
    required this.lines,
  });
}

class TextLine {
  final String text;
  final Rect boundingBox;
  final List elements;
  
  TextLine({
    required this.text,
    required this.boundingBox,
    required this.elements,
  });
}

class TextElement {
  final String text;
  final Rect boundingBox;
  
  TextElement({
    required this.text,
    required this.boundingBox,
  });
}

Document Scanner UI

class DocumentScannerScreen extends StatefulWidget {
  @override
  _DocumentScannerScreenState createState() => _DocumentScannerScreenState();
}

class _DocumentScannerScreenState extends State {
  final TextRecognitionService _textService = TextRecognitionService();
  final ImagePicker _imagePicker = ImagePicker();
  
  File? _imageFile;
  RecognizedText? _recognizedText;
  bool _isProcessing = false;
  
  Future _pickImage(ImageSource source) async {
    final pickedFile = await _imagePicker.pickImage(source: source);
    
    if (pickedFile != null) {
      setState(() {
        _imageFile = File(pickedFile.path);
        _isProcessing = true;
      });
      
      final recognizedText = await _textService.recognizeText(_imageFile!);
      
      setState(() {
        _recognizedText = recognizedText;
        _isProcessing = false;
      });
    }
  }
  
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text('Document Scanner'),
        actions: [
          if (_recognizedText != null)
            IconButton(
              icon: Icon(Icons.copy),
              onPressed: _copyAllText,
              tooltip: 'Copy all text',
            ),
          IconButton(
            icon: Icon(Icons.share),
            onPressed: _recognizedText != null ? _shareText : null,
            tooltip: 'Share text',
          ),
        ],
      ),
      body: Column(
        children: [
          Expanded(
            child: _imageFile == null
                ? _buildPlaceholder()
                : _buildImageView(),
          ),
          _buildActionButtons(),
        ],
      ),
    );
  }
  
  Widget _buildPlaceholder() {
    return Center(
      child: Column(
        mainAxisAlignment: MainAxisAlignment.center,
        children: [
          Icon(
            Icons.document_scanner,
            size: 100,
            color: Colors.grey[400],
          ),
          SizedBox(height: 20),
          Text(
            'Scan a document to extract text',
            style: TextStyle(
              fontSize: 18,
              color: Colors.grey[600],
            ),
          ),
        ],
      ),
    );
  }
  
  Widget _buildImageView() {
    return Stack(
      fit: StackFit.expand,
      children: [
        InteractiveViewer(
          child: Image.file(
            _imageFile!,
            fit: BoxFit.contain,
          ),
        ),
        if (_recognizedText != null)
          CustomPaint(
            painter: TextBlockPainter(
              recognizedText: _recognizedText!,
              imageFile: _imageFile!,
            ),
          ),
        if (_isProcessing)
          Container(
            color: Colors.black54,
            child: Center(
              child: Column(
                mainAxisSize: MainAxisSize.min,
                children: [
                  CircularProgressIndicator(),
                  SizedBox(height: 16),
                  Text(
                    'Processing image...',
                    style: TextStyle(color: Colors.white),
                  ),
                ],
              ),
            ),
          ),
      ],
    );
  }
  
  Widget _buildActionButtons() {
    return Container(
      padding: EdgeInsets.all(16),
      child: Column(
        children: [
          if (_recognizedText != null && !_isProcessing)
            Container(
              height: 150,
              margin: EdgeInsets.only(bottom: 16),
              padding: EdgeInsets.all(12),
              decoration: BoxDecoration(
                color: Colors.grey[100],
                borderRadius: BorderRadius.circular(12),
                border: Border.all(color: Colors.grey[300]!),
              ),
              child: Column(
                crossAxisAlignment: CrossAxisAlignment.start,
                children: [
                  Row(
                    mainAxisAlignment: MainAxisAlignment.spaceBetween,
                    children: [
                      Text(
                        'Extracted Text',
                        style: TextStyle(fontWeight: FontWeight.bold),
                      ),
                      Text(
                        '${_recognizedText!.text.length} characters',
                        style: TextStyle(fontSize: 12, color: Colors.grey),
                      ),
                    ],
                  ),
                  SizedBox(height: 8),
                  Expanded(
                    child: SingleChildScrollView(
                      child: SelectableText(
                        _recognizedText!.text,
                        style: TextStyle(fontSize: 14),
                      ),
                    ),
                  ),
                ],
              ),
            ),
          Row(
            mainAxisAlignment: MainAxisAlignment.spaceEvenly,
            children: [
              ElevatedButton.icon(
                onPressed: () => _pickImage(ImageSource.camera),
                icon: Icon(Icons.camera_alt),
                label: Text('Camera'),
              ),
              ElevatedButton.icon(
                onPressed: () => _pickImage(ImageSource.gallery),
                icon: Icon(Icons.photo_library),
                label: Text('Gallery'),
              ),
              if (_recognizedText != null)
                ElevatedButton.icon(
                  onPressed: _extractInfo,
                  icon: Icon(Icons.analytics),
                  label: Text('Extract Info'),
                ),
            ],
          ),
        ],
      ),
    );
  }
  
  void _copyAllText() {
    if (_recognizedText != null) {
      Clipboard.setData(ClipboardData(text: _recognizedText!.text));
      ScaffoldMessenger.of(context).showSnackBar(
        SnackBar(content: Text('Text copied to clipboard')),
      );
    }
  }
  
  void _shareText() async {
    if (_recognizedText != null) {
      await Share.share(_recognizedText!.text);
    }
  }
  
  void _extractInfo() {
    if (_recognizedText == null) return;
    
    final text = _recognizedText!.text;
    final phones = _textService.extractPhoneNumbers(text);
    final emails = _textService.extractEmails(text);
    final urls = _textService.extractUrls(text);
    
    showDialog(
      context: context,
      builder: (context) => AlertDialog(
        title: Text('Extracted Information'),
        content: Column(
          mainAxisSize: MainAxisSize.min,
          crossAxisAlignment: CrossAxisAlignment.start,
          children: [
            if (phones.isNotEmpty) ...[
              Text('Phone Numbers:', style: TextStyle(fontWeight: FontWeight.bold)),
              Text(phones),
              SizedBox(height: 12),
            ],
            if (emails.isNotEmpty) ...[
              Text('Emails:', style: TextStyle(fontWeight: FontWeight.bold)),
              Text(emails),
              SizedBox(height: 12),
            ],
            if (urls.isNotEmpty) ...[
              Text('URLs:', style: TextStyle(fontWeight: FontWeight.bold)),
              Text(urls),
            ],
            if (phones.isEmpty && emails.isEmpty && urls.isEmpty)
              Text('No contact information found'),
          ],
        ),
        actions: [
          TextButton(
            onPressed: () => Navigator.pop(context),
            child: Text('Close'),
          ),
        ],
      ),
    );
  }
  
  @override
  void dispose() {
    _textService.dispose();
    super.dispose();
  }
}

Barcode Scanning

ML Kit's barcode scanning API supports most standard barcode formats, including QR codes, Data Matrix, PDF-417, and many 1D formats. It's incredibly fast and accurate, making it perfect for inventory management, event check-ins, and payment applications.

Barcode Scanner Implementation

class BarcodeScannerService {
  late BarcodeScanner _barcodeScanner;
  
  BarcodeScannerService() {
    _barcodeScanner = GoogleMlKit.vision.barcodeScanner(
      formats: [
        BarcodeFormat.all,  // Scan all supported formats
      ],
    );
  }
  
  Future> scanBarcodes(InputImage inputImage) async {
    try {
      final barcodes = await _barcodeScanner.processImage(inputImage);
      return barcodes;
    } catch (e) {
      print('Error scanning barcodes: $e');
      return [];
    }
  }
  
  String getBarcodeValueDescription(Barcode barcode) {
    switch (barcode.type) {
      case BarcodeType.wifi:
        final wifi = barcode.value.wifi!;
        return 'WiFi Network: ${wifi.ssid}\nPassword: ${wifi.password}\nType: ${wifi.encryptionType}';
      
      case BarcodeType.url:
        return 'URL: ${barcode.value.url!.url}';
      
      case BarcodeType.email:
        final email = barcode.value.email!;
        return 'Email: ${email.address}\nSubject: ${email.subject ?? "N/A"}\nBody: ${email.body ?? "N/A"}';
      
      case BarcodeType.phone:
        return 'Phone: ${barcode.value.phone!.number}';
      
      case BarcodeType.sms:
        final sms = barcode.value.sms!;
        return 'SMS to: ${sms.phoneNumber}\nMessage: ${sms.message}';
      
      case BarcodeType.geoCoordinates:
        final geo = barcode.value.geoPoint!;
        return 'Location: ${geo.latitude}, ${geo.longitude}';
      
      case BarcodeType.contactInfo:
        final contact = barcode.value.contactInfo!;
        return '''Contact:
Name: ${contact.name?.formattedName ?? "N/A"}
Phone: ${contact.phones.map((p) => p.number).join(", ")}
Email: ${contact.emails.map((e) => e.address).join(", ")}
Organization: ${contact.organization ?? "N/A"}''';
      
      case BarcodeType.product:
        return 'Product Code: ${barcode.displayValue}';
      
      default:
        return 'Data: ${barcode.displayValue}';
    }
  }
  
  Future handleBarcodeAction(BuildContext context, Barcode barcode) async {
    switch (barcode.type) {
      case BarcodeType.url:
        final url = barcode.value.url!.url!;
        if (await canLaunch(url)) {
          await launch(url);
        }
        break;
      
      case BarcodeType.phone:
        final phone = barcode.value.phone!.number!;
        final phoneUrl = 'tel:$phone';
        if (await canLaunch(phoneUrl)) {
          await launch(phoneUrl);
        }
        break;
      
      case BarcodeType.email:
        final email = barcode.value.email!;
        final emailUrl = 'mailto:${email.address}?subject=${email.subject ?? ""}&body=${email.body ?? ""}';
        if (await canLaunch(emailUrl)) {
          await launch(emailUrl);
        }
        break;
      
      case BarcodeType.wifi:
        _showWifiDialog(context, barcode.value.wifi!);
        break;
      
      case BarcodeType.geoCoordinates:
        final geo = barcode.value.geoPoint!;
        final mapUrl = 'https://maps.google.com/?q=${geo.latitude},${geo.longitude}';
        if (await canLaunch(mapUrl)) {
          await launch(mapUrl);
        }
        break;
      
      default:
        // Copy to clipboard for other types
        Clipboard.setData(ClipboardData(text: barcode.displayValue ?? ''));
        ScaffoldMessenger.of(context).showSnackBar(
          SnackBar(content: Text('Copied to clipboard')),
        );
    }
  }
  
  void _showWifiDialog(BuildContext context, BarcodeWifi wifi) {
    showDialog(
      context: context,
      builder: (context) => AlertDialog(
        title: Text('Connect to WiFi?'),
        content: Column(
          mainAxisSize: MainAxisSize.min,
          crossAxisAlignment: CrossAxisAlignment.start,
          children: [
            Text('Network: ${wifi.ssid}'),
            Text('Security: ${wifi.encryptionType}'),
            if (wifi.password != null)
              Text('Password: ${wifi.password}'),
          ],
        ),
        actions: [
          TextButton(
            onPressed: () => Navigator.pop(context),
            child: Text('Cancel'),
          ),
          ElevatedButton(
            onPressed: () {
              // Implement WiFi connection logic
              Navigator.pop(context);
            },
            child: Text('Connect'),
          ),
        ],
      ),
    );
  }
  
  void dispose() {
    _barcodeScanner.close();
  }
}

QR Code Scanner UI

class QRScannerScreen extends StatefulWidget {
  @override
  _QRScannerScreenState createState() => _QRScannerScreenState();
}

class _QRScannerScreenState extends State 
    with SingleTickerProviderStateMixin {
  final BarcodeScannerService _scannerService = BarcodeScannerService();
  CameraController? _cameraController;
  List _barcodes = [];
  bool _isScanning = false;
  late AnimationController _animationController;
  
  @override
  void initState() {
    super.initState();
    _animationController = AnimationController(
      duration: Duration(seconds: 2),
      vsync: this,
    )..repeat();
    _initializeCamera();
  }
  
  @override
  Widget build(BuildContext context) {
    if (_cameraController == null || !_cameraController!.value.isInitialized) {
      return Scaffold(
        body: Center(child: CircularProgressIndicator()),
      );
    }
    
    return Scaffold(
      body: Stack(
        fit: StackFit.expand,
        children: [
          CameraPreview(_cameraController!),
          _buildScannerOverlay(),
          if (_barcodes.isNotEmpty)
            Positioned(
              bottom: 0,
              left: 0,
              right: 0,
              child: _buildResultCard(),
            ),
          Positioned(
            top: 50,
            left: 20,
            child: CircleAvatar(
              backgroundColor: Colors.black54,
              child: IconButton(
                icon: Icon(Icons.close, color: Colors.white),
                onPressed: () => Navigator.pop(context),
              ),
            ),
          ),
          Positioned(
            top: 50,
            right: 20,
            child: CircleAvatar(
              backgroundColor: Colors.black54,
              child: IconButton(
                icon: Icon(
                  _cameraController!.value.flashMode == FlashMode.torch
                      ? Icons.flash_on
                      : Icons.flash_off,
                  color: Colors.white,
                ),
                onPressed: _toggleFlash,
              ),
            ),
          ),
        ],
      ),
    );
  }
  
  Widget _buildScannerOverlay() {
    return Center(
      child: Container(
        width: 250,
        height: 250,
        child: Stack(
          children: [
            // Corner borders
            Positioned(
              top: 0,
              left: 0,
              child: _buildCorner(true, true),
            ),
            Positioned(
              top: 0,
              right: 0,
              child: _buildCorner(true, false),
            ),
            Positioned(
              bottom: 0,
              left: 0,
              child: _buildCorner(false, true),
            ),
            Positioned(
              bottom: 0,
              right: 0,
              child: _buildCorner(false, false),
            ),
            // Scanning line animation
            AnimatedBuilder(
              animation: _animationController,
              builder: (context, child) {
                return Positioned(
                  top: _animationController.value * 250,
                  left: 0,
                  right: 0,
                  child: Container(
                    height: 2,
                    decoration: BoxDecoration(
                      gradient: LinearGradient(
                        colors: [
                          Colors.transparent,
                          Colors.red,
                          Colors.transparent,
                        ],
                      ),
                    ),
                  ),
                );
              },
            ),
          ],
        ),
      ),
    );
  }
  
  Widget _buildCorner(bool isTop, bool isLeft) {
    return Container(
      width: 50,
      height: 50,
      decoration: BoxDecoration(
        border: Border(
          top: isTop
              ? BorderSide(color: Colors.white, width: 4)
              : BorderSide.none,
          bottom: !isTop
              ? BorderSide(color: Colors.white, width: 4)
              : BorderSide.none,
          left: isLeft
              ? BorderSide(color: Colors.white, width: 4)
              : BorderSide.none,
          right: !isLeft
              ? BorderSide(color: Colors.white, width: 4)
              : BorderSide.none,
        ),
      ),
    );
  }
  
  Widget _buildResultCard() {
    final barcode = _barcodes.first;
    
    return Container(
      padding: EdgeInsets.all(20),
      decoration: BoxDecoration(
        color: Colors.white,
        borderRadius: BorderRadius.vertical(top: Radius.circular(20)),
        boxShadow: [
          BoxShadow(
            color: Colors.black26,
            blurRadius: 10,
            offset: Offset(0, -2),
          ),
        ],
      ),
      child: Column(
        mainAxisSize: MainAxisSize.min,
        children: [
          Icon(
            _getIconForBarcodeType(barcode.type),
            size: 40,
            color: Theme.of(context).primaryColor,
          ),
          SizedBox(height: 12),
          Text(
            _getBarcodeTypeLabel(barcode.type),
            style: TextStyle(
              fontSize: 16,
              fontWeight: FontWeight.bold,
            ),
          ),
          SizedBox(height: 8),
          Text(
            _scannerService.getBarcodeValueDescription(barcode),
            textAlign: TextAlign.center,
            style: TextStyle(fontSize: 14),
          ),
          SizedBox(height: 16),
          Row(
            mainAxisAlignment: MainAxisAlignment.spaceEvenly,
            children: [
              OutlinedButton(
                onPressed: () {
                  setState(() {
                    _barcodes.clear();
                  });
                },
                child: Text('Scan Again'),
              ),
              ElevatedButton(
                onPressed: () {
                  _scannerService.handleBarcodeAction(context, barcode);
                },
                child: Text(_getActionLabel(barcode.type)),
              ),
            ],
          ),
        ],
      ),
    );
  }
  
  IconData _getIconForBarcodeType(BarcodeType type) {
    switch (type) {
      case BarcodeType.wifi:
        return Icons.wifi;
      case BarcodeType.url:
        return Icons.link;
      case BarcodeType.email:
        return Icons.email;
      case BarcodeType.phone:
        return Icons.phone;
      case BarcodeType.sms:
        return Icons.sms;
      case BarcodeType.geoCoordinates:
        return Icons.location_on;
      case BarcodeType.contactInfo:
        return Icons.contact_phone;
      case BarcodeType.product:
        return Icons.shopping_cart;
      default:
        return Icons.qr_code;
    }
  }
  
  String _getBarcodeTypeLabel(BarcodeType type) {
    switch (type) {
      case BarcodeType.wifi:
        return 'WiFi Network';
      case BarcodeType.url:
        return 'Website URL';
      case BarcodeType.email:
        return 'Email Address';
      case BarcodeType.phone:
        return 'Phone Number';
      case BarcodeType.sms:
        return 'SMS Message';
      case BarcodeType.geoCoordinates:
        return 'Location';
      case BarcodeType.contactInfo:
        return 'Contact Information';
      case BarcodeType.product:
        return 'Product Code';
      default:
        return 'Barcode Data';
    }
  }
  
  String _getActionLabel(BarcodeType type) {
    switch (type) {
      case BarcodeType.wifi:
        return 'Connect';
      case BarcodeType.url:
        return 'Open';
      case BarcodeType.email:
        return 'Compose';
      case BarcodeType.phone:
        return 'Call';
      case BarcodeType.sms:
        return 'Send SMS';
      case BarcodeType.geoCoordinates:
        return 'View Map';
      case BarcodeType.contactInfo:
        return 'Add Contact';
      default:
        return 'Copy';
    }
  }
}

Image Labeling

Image labeling identifies objects, locations, activities, animal species, products, and more in images. It's useful for content moderation, image search, and accessibility features.

Image Labeling Service

class ImageLabelingService {
  late ImageLabeler _imageLabeler;
  
  ImageLabelingService({bool useCloudModel = false}) {
    if (useCloudModel) {
      // Cloud model provides more detailed results
      _imageLabeler = GoogleMlKit.vision.imageLabeler(
        ImageLabelerOptions(confidenceThreshold: 0.7),
      );
    } else {
      // On-device model for offline use
      _imageLabeler = GoogleMlKit.vision.imageLabeler(
        ImageLabelerOptions(confidenceThreshold: 0.5),
      );
    }
  }
  
  Future> labelImage(File imageFile) async {
    try {
      final inputImage = InputImage.fromFile(imageFile);
      final labels = await _imageLabeler.processImage(inputImage);
      
      // Sort by confidence
      labels.sort((a, b) => b.confidence.compareTo(a.confidence));
      
      return labels;
    } catch (e) {
      print('Error labeling image: $e');
      return [];
    }
  }
  
  List getCategories(List labels) {
    final categories = {};
    
    for (final label in labels) {
      // Extract broad categories
      if (label.label.contains('Food')) categories.add('Food');
      if (label.label.contains('Animal')) categories.add('Animals');
      if (label.label.contains('Person') || label.label.contains('People')) {
        categories.add('People');
      }
      if (label.label.contains('Nature') || label.label.contains('Plant')) {
        categories.add('Nature');
      }
      if (label.label.contains('Vehicle') || label.label.contains('Car')) {
        categories.add('Transportation');
      }
      if (label.label.contains('Building') || label.label.contains('Architecture')) {
        categories.add('Architecture');
      }
    }
    
    return categories.toList();
  }
  
  double calculateContentSafety(List labels) {
    // Simple content safety check based on labels
    final unsafeKeywords = [
      'Violence', 'Weapon', 'Blood', 'Adult', 'Explicit'
    ];
    
    double safetyScore = 1.0;
    
    for (final label in labels) {
      for (final keyword in unsafeKeywords) {
        if (label.label.toLowerCase().contains(keyword.toLowerCase())) {
          safetyScore -= (label.confidence * 0.3);
        }
      }
    }
    
    return safetyScore.clamp(0.0, 1.0);
  }
  
  void dispose() {
    _imageLabeler.close();
  }
}

Pose Detection

Pose detection identifies the positions of key body joints in images and video. It's perfect for fitness apps, games, and health monitoring applications.

Pose Detection Implementation

class PoseDetectionService {
  late PoseDetector _poseDetector;
  
  PoseDetectionService({bool useAccurateModel = false}) {
    _poseDetector = GoogleMlKit.vision.poseDetector(
      PoseDetectorOptions(
        mode: useAccurateModel 
            ? PoseDetectionMode.accurate 
            : PoseDetectionMode.stream,
        model: PoseDetectionModel.base,
        enableClassification: true,
        enableSmoothing: true,
      ),
    );
  }
  
  Future> detectPoses(InputImage inputImage) async {
    try {
      final poses = await _poseDetector.processImage(inputImage);
      return poses;
    } catch (e) {
      print('Error detecting poses: $e');
      return [];
    }
  }
  
  Map analyzePose(Pose pose) {
    final landmarks = pose.landmarks;
    final analysis = {};
    
    // Calculate angles between joints
    if (landmarks.containsKey(PoseLandmarkType.leftShoulder) &&
        landmarks.containsKey(PoseLandmarkType.leftElbow) &&
        landmarks.containsKey(PoseLandmarkType.leftWrist)) {
      
      final shoulder = landmarks[PoseLandmarkType.leftShoulder]!;
      final elbow = landmarks[PoseLandmarkType.leftElbow]!;
      final wrist = landmarks[PoseLandmarkType.leftWrist]!;
      
      final angle = _calculateAngle(
        shoulder.position,
        elbow.position,
        wrist.position,
      );
      
      analysis['leftArmAngle'] = angle;
    }
    
    // Check if person is standing/sitting
    if (landmarks.containsKey(PoseLandmarkType.leftHip) &&
        landmarks.containsKey(PoseLandmarkType.leftKnee)) {
      
      final hip = landmarks[PoseLandmarkType.leftHip]!;
      final knee = landmarks[PoseLandmarkType.leftKnee]!;
      
      final verticalDiff = (hip.position.y - knee.position.y).abs();
      analysis['isStanding'] = verticalDiff > 50 ? 1.0 : 0.0;
    }
    
    return analysis;
  }
  
  double _calculateAngle(Point a, Point b, Point c) {
    final radians = atan2(c.y - b.y, c.x - b.x) - atan2(a.y - b.y, a.x - b.x);
    var angle = radians * 180 / pi;
    
    if (angle < 0) {
      angle += 360;
    }
    
    return angle;
  }
  
  bool isExerciseFormCorrect(Pose pose, String exerciseType) {
    final analysis = analyzePose(pose);
    
    switch (exerciseType) {
      case 'pushup':
        final armAngle = analysis['leftArmAngle'] ?? 0;
        return armAngle > 160 && armAngle < 200;
      
      case 'squat':
        // Check knee and hip angles
        return true; // Implement specific logic
      
      default:
        return true;
    }
  }
  
  void dispose() {
    _poseDetector.close();
  }
}

Selfie Segmentation

Selfie segmentation separates the person from the background, enabling virtual backgrounds and other creative effects in real-time.

Selfie Segmentation Implementation

class SelfieSegmentationService {
  late SelfieSegmenter _segmenter;
  
  SelfieSegmentationService() {
    _segmenter = GoogleMlKit.vision.selfieSegmenter(
      SelfieSegmenterOptions(
        mode: SegmenterMode.stream,
        enableRawSizeMask: true,
      ),
    );
  }
  
  Future segmentSelfie(InputImage inputImage) async {
    try {
      final mask = await _segmenter.processImage(inputImage);
      return mask;
    } catch (e) {
      print('Error segmenting selfie: $e');
      return null;
    }
  }
  
  Uint8List applyVirtualBackground(
    Uint8List originalImage,
    SegmentationMask mask,
    Uint8List backgroundImage,
  ) {
    // Implementation for applying virtual background
    // This would typically involve:
    // 1. Decode both images
    // 2. Apply mask to separate foreground/background
    // 3. Composite images
    // 4. Encode result
    
    // Placeholder implementation
    return originalImage;
  }
  
  void dispose() {
    _segmenter.close();
  }
}

Best Practices

Performance Optimization

• Use appropriate image sizes: Resize images before processing to balance accuracy and speed
• Batch processing: Process multiple images together when possible
• Cache results: Store processed results to avoid redundant processing
• Use streaming mode: For real-time camera processing, use stream mode for better performance

Memory Management

class MLKitMemoryManager {
  static final Map _detectorCache = {};
  
  static T getOrCreateDetector(
    String key,
    T Function() creator,
  ) {
    if (!_detectorCache.containsKey(key)) {
      _detectorCache[key] = creator();
    }
    return _detectorCache[key] as T;
  }
  
  static void disposeDetector(String key) {
    if (_detectorCache.containsKey(key)) {
      final detector = _detectorCache[key];
      if (detector is FaceDetector) detector.close();
      if (detector is TextRecognizer) detector.close();
      if (detector is BarcodeScanner) detector.close();
      // Add other detector types
      
      _detectorCache.remove(key);
    }
  }
  
  static void disposeAll() {
    _detectorCache.forEach((key, detector) {
      disposeDetector(key);
    });
    _detectorCache.clear();
  }
}

Error Handling

class MLKitErrorHandler {
  static String getErrorMessage(dynamic error) {
    if (error.toString().contains('InputImage')) {
      return 'Invalid image format. Please try another image.';
    } else if (error.toString().contains('permission')) {
      return 'Camera permission required for this feature.';
    } else if (error.toString().contains('model')) {
      return 'ML model initialization failed. Please restart the app.';
    } else {
      return 'An unexpected error occurred. Please try again.';
    }
  }
  
  static void logError(String feature, dynamic error, StackTrace? stack) {
    print('[ML Kit - $feature] Error: $error');
    if (stack != null) {
      print('Stack trace: $stack');
    }
    
    // Send to analytics/crash reporting
    // FirebaseCrashlytics.instance.recordError(error, stack);
  }
}

Real-World Applications

1. Smart Document Scanner

Combine text recognition with image processing to create a powerful document scanner that automatically detects documents, corrects perspective, enhances text, and extracts information.

2. Fitness Trainer App

Use pose detection to create a virtual fitness trainer that provides real-time feedback on exercise form, counts repetitions, and tracks progress.

3. Accessibility Assistant

Leverage multiple ML Kit APIs to build an accessibility app that describes scenes (image labeling), reads text aloud (OCR), and identifies objects for visually impaired users.

4. AR Face Filters

Use face detection with landmarks and contours to create Instagram-style face filters with real-time effects and animations.

5. Inventory Management

Build a barcode scanning solution for inventory tracking with batch scanning, offline support, and integration with backend systems.

Conclusion

Google ML Kit Vision APIs provide powerful computer vision capabilities that can transform your Flutter applications. From face detection to text recognition, these APIs offer production-ready solutions that work offline, process in real-time, and maintain user privacy.

Key Takeaways

• ML Kit offers comprehensive vision APIs for common use cases
• On-device processing ensures privacy and offline functionality
• Easy integration with Flutter through google_ml_kit package
• Real-time performance suitable for camera-based applications
• Flexible options between accuracy and speed
• Cross-platform consistency with unified API

Next Steps

1. Start Simple: Begin with a single API like text recognition or barcode scanning
2. Combine APIs: Create more powerful features by combining multiple vision APIs
3. Optimize Performance: Profile and optimize for your specific use case
4. Handle Edge Cases: Test with various lighting conditions and device types
5. Deploy with Confidence: ML Kit's proven models are production-ready

Related Articles

TensorFlow Lite in Flutter

Build custom ML models for Flutter

Google Gemini Integration

Add conversational AI to Flutter apps

AI Chatbots with LangChain

Build intelligent conversational interfaces