Building an Automated Face Recognition Pipeline with Python and OpenCV

The Problem: Too Many Photos, No Way to Sort Them by Person

Anyone who's tried to organize a large folder of event or family photos knows the pain: hundreds of images, no consistent naming, and no easy way to pull out "every photo that has this specific person in it." Doing this manually means opening each image and eyeballing it. I wanted to automate it — not with a cloud service, but with a self-contained Python pipeline I could run locally.

That project became Automated Image Filtering Using Face Recognition, built with Python, OpenCV, InsightFace, and NumPy. What made it click conceptually was framing it not as "a computer vision script" but as an RPA-style pipeline: input, processing, decision-making, output — the same mental model I use for the RPA bots I build professionally, just applied to image data instead of business documents.

Why the RPA Framing Matters

RPA pipelines are usually described as: capture an input, apply deterministic or rule-based processing, make a decision based on some criteria, and produce a structured output — ideally with minimal human intervention and a predictable, auditable flow. That framing maps cleanly onto a face-matching pipeline:

Input: raw folder of unsorted images
    ↓
Processing: face detection + embedding extraction (OpenCV + InsightFace)
    ↓
Decision-making: similarity comparison against reference profiles
    ↓
Output: images copied/moved into structured, per-person folders

Thinking about it this way — rather than as a one-off script — pushed me to design it with the same discipline I'd apply to a production bot: clear stage boundaries, predictable failure handling, and structured output rather than console prints.

Stage 1: Input — Building Reference Profiles

Before the pipeline can match anything, it needs reference face embeddings for each person it should recognize. I built a small setup step that takes a folder of clearly labeled reference photos (one or more per person) and extracts their face embeddings using InsightFace:

import os
import numpy as np
from insightface.app import FaceAnalysis

face_app = FaceAnalysis(name="buffalo_l")
face_app.prepare(ctx_id=0, det_size=(640, 640))

def build_reference_profiles(reference_dir: str) -> dict:
    """Build a dict of {person_name: [embedding, embedding, ...]} from labeled folders."""
    profiles = {}
    for person_name in os.listdir(reference_dir):
        person_path = os.path.join(reference_dir, person_name)
        if not os.path.isdir(person_path):
            continue

        embeddings = []
        for img_file in os.listdir(person_path):
            img_path = os.path.join(person_path, img_file)
            img = cv2.imread(img_path)
            faces = face_app.get(img)
            if faces:
                embeddings.append(faces[0].embedding)

        if embeddings:
            profiles[person_name] = embeddings
    return profiles

Each person can have multiple reference images because a single photo rarely captures enough variation in lighting or angle to match reliably against real-world input images.

Stage 2: Processing — Detecting Faces in the Input Set

For every image in the unsorted input folder, the pipeline detects all faces present and extracts an embedding for each one:

import cv2

def extract_faces_from_image(img_path: str) -> list:
    """Detect all faces in an image and return their embeddings + bounding boxes."""
    img = cv2.imread(img_path)
    if img is None:
        return []

    faces = face_app.get(img)
    return [
        {"embedding": f.embedding, "bbox": f.bbox, "det_score": f.det_score}
        for f in faces
    ]

I filter out low-confidence detections early (det_score below a threshold) so blurry or partial faces don't pollute the matching stage with noisy embeddings.

Stage 3: Decision-Making — Matching Against Reference Profiles

This is the heart of the pipeline. For each detected face, I compare its embedding against every reference profile using cosine similarity, and assign the image to a person if the similarity clears a threshold:

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

SIMILARITY_THRESHOLD = 0.45

def match_face_to_profile(embedding: np.ndarray, profiles: dict) -> str | None:
    """Return the best-matching person name, or None if no match clears the threshold."""
    best_match = None
    best_score = 0.0

    for person_name, ref_embeddings in profiles.items():
        # Compare against all reference embeddings for this person, take the max
        scores = [cosine_similarity(embedding, ref) for ref in ref_embeddings]
        person_best = max(scores)

        if person_best > best_score:
            best_score = person_best
            best_match = person_name

    return best_match if best_score >= SIMILARITY_THRESHOLD else None

Tuning SIMILARITY_THRESHOLD was mostly empirical — too low and unrelated people get matched together; too high and legitimate matches get missed under different lighting. I settled on a value after testing against a mixed batch of known-correct and known-incorrect pairs and checking where false positives started creeping in.

Stage 4: Output — Structured, Auditable Results

The final stage copies (never moves, to avoid destructive mistakes) matched images into per-person output folders, and logs anything that didn't clear the threshold into an "unmatched" folder for manual review:

import shutil

def route_image(img_path: str, matched_person: str | None, output_dir: str):
    """Copy image into the correct output folder based on match result."""
    target_folder = matched_person if matched_person else "unmatched"
    dest_dir = os.path.join(output_dir, target_folder)
    os.makedirs(dest_dir, exist_ok=True)
    shutil.copy2(img_path, os.path.join(dest_dir, os.path.basename(img_path)))

def process_pipeline(input_dir: str, reference_dir: str, output_dir: str):
    profiles = build_reference_profiles(reference_dir)

    for img_file in os.listdir(input_dir):
        img_path = os.path.join(input_dir, img_file)
        faces = extract_faces_from_image(img_path)

        if not faces:
            route_image(img_path, None, output_dir)
            continue

        # Route based on the highest-confidence face detected
        best_face = max(faces, key=lambda f: f["det_score"])
        matched_person = match_face_to_profile(best_face["embedding"], profiles)
        route_image(img_path, matched_person, output_dir)

Keeping an explicit "unmatched" bucket instead of silently discarding low-confidence results was a deliberate RPA-style decision — it mirrors how I handle low-confidence cases in production bots at Avent IQ: route it to a review queue rather than let it fail invisibly.

Lessons from Treating a CV Script Like an RPA Pipeline

1. Non-destructive operations first. Copying instead of moving files meant a bug in the matching logic never risked losing the original unsorted photos — the same principle applies to any automation touching production data.
2. Threshold-based decisions need an escape hatch. Just like the confidence-based routing I use in real automation work, a similarity threshold needs a place for uncertain cases to land, not just a binary pass/fail.
3. Multiple reference samples per entity beat one "perfect" sample. One reference photo is fragile against real-world variation; a handful of samples per person made matching noticeably more reliable.
4. Separate concerns cleanly. Keeping detection, embedding, matching, and file routing as distinct functions made it far easier to debug which stage was responsible when a match went wrong — the same modularity that makes RPA bots maintainable.

Interested in Similar Automation Work?

If you're working on a Python automation, computer vision, or RPA-style pipeline and want to talk through the design, I'd be happy to help. Reach me at rishabnishad22@gmail.com or on WhatsApp, or check out more of my projects at rishab-nishad.vercel.app.