On-Device Face Verification

The problem

Login, attendance, and logout are the app's trust-critical actions, and they were protected by nothing that proves who is holding the phone. Proxy attendance ("buddy punching") breaks the product's core promise. The fix is 1:1 face verification: is the person at the camera the one enrolled employee for this device?

The constraints made it interesting. Field staff work in network dead zones, so the gate had to be offline-first and never strand a user. The devices are low-end Androids, so model size, load time, and inference latency were real budgets. The environments are clinics, classrooms, and outdoor sites: cluttered scenes, posters with faces on them, bystanders, bad lighting. The users are diverse, and a biometric gate that locks out people with narrow eyes or beards is a failed feature. And biometrics are sensitive, so the privacy bar was strict on-device processing.

There's a continuity worth noting: the iOS app this gate ships into is one I built myself, in my first internship at the same company: a Flutter/Dart port (with native Swift) of their native-Java Android app, whose own hard problem was keeping a continuous field-tracking app alive as an iOS background process with zero-data-loss offline sync and tamper-resistant logic. Both internships' work runs in production today for the same 650+ field users.

Interactive: enroll, verify, and try to fool it

A live model of the verification pipeline. Enroll a user, verify them, then send an impostor and a photo-spoof at the gate and watch which layer stops each one. The thresholds shown (0.38 / 0.35 / 0.33) are the real server-pushed triple; similarity values are illustrative of the harness-scale separation (genuine ≈ 0.47, impostor ≈ 0.07), not production accuracy claims.

How it works

Three independent security layers, each able to reject a frame on its own: quality, liveness, and identity. The per-frame pipeline:

Camera frame ─▶ YUV_420_888 → ARGB (pure-Java BT.601), rotate, mirror
   ─▶ ML Kit detect + landmarks + head pose
   ─▶ ROI + largest-face primary-subject pick
   ─▶ quality gate (size / brightness / blur / landmarks / pose)
   ─▶ ArcFace 5-point aligned 112×112 crop
   ─▶ TFLite embed → L2-normalise
   ─▶ [enroll]  outlier-trim + store multi-template
      [verify]  cosine vs every stored vector → dual-threshold accept

Enrollment: a multi-template, not a centroid

Enrollment captures 5 frontal frames, embeds each, and iteratively trims outliers (recompute the centroid, drop the least-similar frame while consistency is below a floor, never below 3 survivors). If a consistent set can't be reached, the user re-enrolls, because the system refuses to silently store a low-confidence template. The surviving individual vectors are stored, not their average, because the matcher needs them.

Verification: best-of-N×M with a corroborated accept

Verification collects up to 3 frontal probe embeddings while the liveness challenge runs, then scores every (probe × stored-vector) cosine pair. The accept rule is dual-path: one very strong match (≥ 0.38), or a good match (≥ 0.35) corroborated by a second independent good pair (≥ 0.33). This way a single fluke high score can't authenticate an impostor, and one bad frame can't lock out a genuine user. All three thresholds are server-controlled, clamped, cached, and stamped with the model version.

Liveness, independent of identity

An active "move your head" challenge: a yaw-driven state machine that requires starting roughly frontal and then turning past a threshold within a timeout. A flat photo fails it; and passing the challenge implies nothing about identity, since the two layers are deliberately independent.

The server control plane (that never bricks attendance)

A per-device/account mode (off / on / on + event-selfie) and the three thresholds arrive in one compact string over a 2-second-timeout call. Resolution is online → cached → disabled: a network blip in the field degrades gracefully instead of stranding a worker at a locked gate.

The model story: evaluate, migrate, optimize

The incumbent embedder was a five-year-old FaceNet (160×160 input, 128-d output, 23.8 MB). I evaluated MobileFaceNet, ArcFace R50/R100 (an accuracy ceiling, but too heavy for low-end phones), and EdgeFace against an offline 1:1 verification protocol I authored as a standalone eval brief: ROC/EER and TAR@FAR on CelebA/LFW pairs, plus a second, deployment-faithful layer that simulates the real multi-template decision rule. The brief also calls out the classic traps: every model needs its own native preprocessing (a wrong channel order or normalization produces valid-looking-but-wrong embeddings), and thresholds never transfer across models.

The migration to MobileFaceNet (InsightFace w600k_mbf, 112×112, 512-d) came with three optimization calls, each with a root-caused rejection:

• ONNX Runtime: rejected. The Android native library is ~26 MB per ABI; integrating it to run the int8 ONNX directly would have bloated the APK more than the model it served.
• int8 quantization: rejected. Dynamic-range int8 from onnx2tf collapsed the network: every face mapped to the same vector (genuine = impostor = 1.0). Caught before ship because I validated against the fp32 reference instead of the already-degraded int8 output.
• float16 TFLite: shipped. ~6.86 MB (≈3.5× smaller than FaceNet's asset), running delegate-free on the stock CPU interpreter the app already bundled. A fidelity harness showed the fp16 model reproduces the fp32 ONNX embeddings to cosine 1.0000 on proof images.

The hand-written aligner

ArcFace alignment must land 5 facial landmarks on a canonical 112×112 template, but Android's Matrix.setPolyToPoly caps at 4 point correspondences. So I implemented the closed-form least-squares Umeyama similarity transform (rotation + uniform scale + translation) by hand in Java, working from centroids, scatter, and cross-covariance terms to recover s·cosθ and s·sinθ directly, and verified it against skimage.SimilarityTransform and OpenCV to 1e-13 in a dedicated harness.

Design decisions & trade-offs

• Inclusion is an engineering requirement, not a nicety. Early gates rejected real users: ML Kit reports persistently low eye-open probability for naturally narrow-eyed people, so the blink/eye-open checks were removed (opaque sunglasses are caught by the missing eye landmark instead). The mouth-landmark rule converged 3-of-3 → 1-of-3 → 2-of-3: strict rejected beards and round faces, loose let a hand-over-mouth spoof through, 2-of-3 is the measured midpoint. The rationale is encoded in source comments so it can't regress silently.
• No lockout, by design. A sliding-window tracker reports "too many attempts" telemetry after 3 failures in 10 minutes, but never locks the user out, because stranding a genuine field worker is worse than the attack it would prevent.
• ROI + largest-face instead of "reject if two faces". Posters, wall photos, and passers-by constantly trip the naive rule. Faces outside a central oval region are dropped and the largest survivor is taken as the phone-holder. The ROI maps exactly to the on-screen guide oval, so the gate and the visual guide can never drift apart.
• Silent failures are the enemy. A stored biometric is a corruption hazard across model changes, so a template must pass six invalidation gates (format, schema version, model-compat version, owner identity, pipeline recipe, and a model byte-size check as a backstop for "forgot to bump the version"). The threshold cache is stamped with the model version so a stale FaceNet-era 0.70 threshold can't survive a swap and silently reject every ~0.47 MobileFaceNet match.
• Races made structurally impossible. The whole pipeline runs on one single-threaded executor with a pull-based camera (exactly one frame in flight), so embedding state needs no locking at all. The concurrency model was chosen so data races can't exist, rather than being carefully avoided.
• Privacy by construction. Nothing in the ML path emits an image or an embedding. In event-selfie mode, a downscaled frame is written to app-private storage only on a successful verify, and only a file path leaves the activity.

Honest scope

• The accuracy evidence is harness-scale, not production-benchmark scale: embedding-fidelity cosine 1.0000 and the genuine ≈ 0.47 / impostor ≈ 0.07 separation come from verification harnesses on proof images. I don't quote a measured MobileFaceNet-vs-FaceNet accuracy delta because I haven't recorded one.
• The shipped threshold triple (0.38 / 0.35 / 0.33) is a conservative starting point pending on-device re-tuning against real users, not a FAR-anchored, validated operating point.
• Per-model benchmark tables, on-device latency, and model-load times are gaps I know about and haven't measured yet. They'd make the story stronger, and I won't invent them.