Delivered remotely for a business based in Perth, Australia. Names withheld by agreement.
Project Overview
Industry: Healthcare — self-administered injectable medication, patient-facing accessibility and adherence.
Type of solution: A native iOS application with a bespoke on-device computer vision measurement pipeline, a medication scheduling and reminder system, a local health log with export, and multi-profile support. No backend, no network layer.
Business context: Self-administered injectable therapy asks a patient to perform a precise measurement, unaided, in their own home, on a schedule, indefinitely. The measurement is visual and the scale is small. Many of the people doing it have impaired vision, reduced dexterity, or tremor — frequently as a consequence of the same condition the medication treats. The conventional aids are a magnifier or another person, and neither is available three times a day for years.
Alongside the measurement problem sits an evidence problem. Adherence is largely invisible: a clinician reviewing therapy has whatever the patient remembers or wrote down. There is no record of what was actually drawn, or when.
General users: Patients self-administering injectable medication; family members or caregivers managing one or more patient profiles on a shared device; and clinicians as the recipients of exported records.
General purpose: To turn an unaided visual judgement into a verified, spoken, logged and evidenced one — without asking the patient to create an account, trust a cloud service, or own any hardware beyond the phone they already have.
The Business Challenge
The measurement has to be right. This is not a scanning app where a wrong result is an inconvenience. The output is a dose figure a patient may act on medically, on a scale where a single unit matters. There is no comfortable margin for error and no server to correct a bad result afterwards.
The subject matter is hostile to computer vision. The object being measured is a small, transparent, curved cylinder containing a clear liquid, held by hand, photographed at close focus, under whatever lighting exists in a kitchen or a bathroom at six in the morning. Glare, reflection and refraction attack exactly the visual properties any measurement has to rely on.
The users are, by definition, the hardest users. The population that most needs a camera to read a dose is the population least able to hold a phone steady over a small object — low vision, tremor, neuropathy, arthritis. The application had to succeed with poor framing, unsteady hands and imperfect light, rather than demanding a clean capture.
One frame is never enough. Any individual video frame can be blurred, glare-struck or mid-autofocus. A system that trusts the first frame that parses successfully will confidently produce wrong answers.
A number alone is not a safe interaction. Showing a measurement is the easy part. What happens when it is wrong, how the patient corrects it, whether they are warned when it contradicts their prescription, and whether an unconfirmed reading can quietly become a record — those decisions carry more clinical weight than the measurement itself.
Health data raises the stakes on every architectural choice. Dose history, blood-sugar values, dates of birth and photographs are all sensitive. Any transmission, any account, any third-party service becomes a compliance surface and a trust question for a user who is already being asked to point a camera at their medication.
There is no infrastructure to fall back on. With no backend, there is no sync, no server-side correction, no remote diagnostics and no telemetry. Everything has to work correctly on the device, first time, with no way to patch bad data after the fact.
Our Approach
Solve the measurement deterministically, not probabilistically. The device being measured has fixed, known physical geometry and high-contrast landmarks. That makes the problem tractable without machine learning: locate the landmarks by direct pixel classification, measure the distance to the indicator position, and convert through the known geometry. We chose this over a trained model deliberately. It produces an explainable result rather than a confidence score, it needs no training data, it ships no model, it adds no size, and it behaves identically on every device — all of which matter more in a safety-relevant health context than the flexibility a model would have bought.
Reject bad input before analysing it. Three cheap gates run before the expensive scan. The gyroscope is sampled continuously and analysis is skipped entirely while the phone is moving. Frame sharpness is computed and blurred frames are discarded. Average brightness is measured and unusable lighting is rejected with a message that says so. Filtering at the source removes an entire class of wrong answers that would otherwise have to be caught downstream.
Reach a decision across frames, not within one. Results accumulate across many sampled frames. Outliers are trimmed against the running average, and once a specific value has repeated often enough it wins outright over the arithmetic mean — the right statistic for a discrete, quantised measurement. Capture stops as soon as agreement is genuine rather than after a fixed timer.
Build the algorithm a way to show its work. A parallel diagnostic pipeline renders the detected landmarks, scan geometry and measurement lines directly onto the analysed frame. For a vision system that has to be tuned against real household lighting rather than a lab, being able to see what the algorithm believes it is looking at is the difference between engineering and guesswork.
Tell the user what actually went wrong. Glare, misalignment and poor lighting are distinguished and produce different guidance. "There appears to be glare — try a different angle" resolves a failed scan; "scan failed" does not.
Use the schedule the patient already gave us. Onboarding captures a full weekly dose schedule. That schedule is then put to work: the measured dose is cross-checked against the prescribed value for that day and time slot, and a mismatch produces a spoken warning and an explicit choice — go back, adjust, or log anyway. The application knows what should have happened, and says so, without ever refusing to record what did.
Never insist the machine is right. Every measurement can be overridden through a large-type picker with discrete up and down controls, designed for low vision and imprecise touch. Readings must be actively confirmed within a short window or they expire rather than silently becoming records.
Choose offline as a feature. No backend, no account, no analytics, no crash reporting, no cloud storage. Health data never leaves the device except through an export the patient explicitly initiates and explicitly shares, with the temporary file removed afterwards. This removes an entire compliance and trust surface rather than mitigating it.
The Solution
Camera-based dose measurement. A live capture session with autofocus handling, torch control and continuous frame sampling, feeding a bespoke pixel-analysis pipeline that locates the device's physical landmarks and converts pixel geometry into a dose value.
Quality gating and stabilisation. Gyroscope motion detection, sharpness analysis and brightness measurement, each capable of rejecting a frame before analysis begins, with targeted on-screen guidance for each failure mode.
Multi-frame consensus. Accumulated results across frames with outlier trimming and modal selection, terminating capture when agreement is reached rather than on a timer.
Spoken dose verification. The measured value is cross-checked against the patient's prescribed schedule for that day and time; on a mismatch the application speaks a warning aloud and offers an explicit three-way choice rather than silently accepting or silently blocking.
Confirmation window. Readings must be confirmed within a short countdown; an unconfirmed reading expires rather than becoming a record by default.
Manual adjustment. A large-type value picker with discrete increment and decrement controls, sized and spaced for impaired vision and reduced dexterity.
Guided schedule setup. A step-by-step wizard capturing personal details, whether the dose is the same every day, which days apply, which times of day apply, and the prescribed units for each slot — producing a full weekly schedule of morning, afternoon and evening positions.
Intelligent reminders. Local notifications are derived from the schedule and the existing reading history together, so a reminder exists only where a dose is genuinely outstanding. Logging a reading immediately rebuilds the entire pending set. A missed-dose check on app open compares the current time against per-slot cut-offs and prompts a check-in.
Photographed reading log. Each reading is stored with its date, time slot, measured units, an optional blood-sugar value and the captured image — so a record can be reviewed visually, not just numerically.
Calendar history. A month view with per-day indicators and month and year navigation, opening into a per-day review of every reading with its photograph and detail.
Date-range export. CSV and PDF export across a selected date range, delivered through the system share sheet for email, messaging or printing ahead of a clinical appointment, with the temporary file deleted once sharing completes.
Multiple local profiles. Create, edit, switch and delete patient profiles on one device, supporting a caregiver managing more than one person.
Support content. A tips carousel covering capture technique, an FAQ, terms of service and privacy policy, and direct contact options.
Key Features
Deterministic on-device dose measurement Physical landmarks located by direct pixel classification and converted through known geometry — explainable, model-free, identical on every device, and entirely offline.
Motion, sharpness and brightness gating Three cheap pre-checks reject unusable frames before analysis runs, eliminating a class of wrong results at the source rather than filtering them afterwards.
Multi-frame consensus with modal selection Agreement is built across many frames with outlier trimming, and the most-repeated value is preferred over the average — the correct statistic for a discrete measurement.
Prescribed-versus-measured cross-check with spoken warning The application knows the prescribed dose for that day and time, compares it against what was measured, and speaks a warning on a mismatch while still allowing the patient to record what actually happened.
Accessibility-first result interaction Large-type display, spoken warnings where safety is at stake, and a manual override with discrete large controls designed for low vision and imprecise touch.
Confirmation window with expiry Readings must be actively confirmed within a countdown, so an unreviewed measurement never becomes a silent medical record.
State-derived reminder scheduling Notifications are rebuilt from the schedule and the reading history together on every change, so patients are reminded only about doses genuinely outstanding.
Photographed, exportable reading history Every reading keeps its own image alongside the numbers, reviewable in a calendar and exportable as CSV or PDF for a clinical appointment.
Multiple local patient profiles Full profile management on a single device, so a caregiver can manage more than one person without separate installations.
Fully offline architecture No backend, no account, no analytics, no third-party services holding health data. Information leaves the device only through an export the user initiates and shares themselves.
Technical Architecture
Presentation layer. A native UIKit application with storyboard-driven navigation, built on a shared base view controller providing navigation chrome, inter-screen parameter passing and export helpers, wrapped in a custom slide-menu container. One screen is implemented in the modern declarative UI framework and hosted inside the UIKit stack.
Capture layer. Camera session management with device configuration, autofocus and torch control, continuous video frame sampling and still capture, plus lifecycle handling so sessions start and stop correctly with navigation.
Measurement layer. An image-processing extension that operates directly on raw bitmap buffers: colour-space-aware context creation, per-pixel classification, landmark detection, geometric conversion, and annotated diagnostic rendering. Analysis is dispatched to background queues with re-entrancy protection, keeping the preview responsive.
Sensor layer. Continuous high-frequency gyroscope sampling providing a live motion state that gates whether analysis is attempted at all.
Consensus layer. Per-frame results accumulated in the scan controller with outlier trimming, modal and mean computation, progress reporting, and termination logic that stops capture on agreement or on a watchdog timeout.
Persistence layer. A local object database with two entities — the patient profile carrying the weekly schedule, and the reading carrying date, time slot, measured units, optional blood-sugar value and the captured image — accessed through a single shared helper with dictionary-based read and write methods.
Scheduling layer. A notification service that reads the schedule and the reading history together and rebuilds the entire pending reminder set on every change, with foreground and background notification handling.
Export layer. Manual CSV construction and PDF rendering from the on-screen view, delivered through the system share sheet with temporary file cleanup on completion.
Commerce layer. Subscription entitlement checking and transaction observation through the platform's in-app purchase framework.
Flow: Camera frames → motion, sharpness and brightness gating → per-pixel landmark detection → geometric conversion to dose → multi-frame consensus → confirmation and manual override → cross-check against prescribed schedule → local persistence with image → reminder set rebuild → calendar review and CSV/PDF export
Technology Stack
| Category | Technology |
|---|---|
| Platform | Native iOS, Swift |
| Primary UI | UIKit with storyboard-driven navigation and a custom slide-menu container |
| Secondary UI | SwiftUI, hosted within the UIKit stack |
| Camera and capture | AVFoundation — capture session, video data output, photo output, focus and torch control |
| Image analysis | Core Graphics raw bitmap contexts and direct pixel buffer access; Core Image for filters and feature detection |
| Motion sensing | Core Motion — gyroscope sampling for stability gating |
| Local persistence | Core Data, device-local only, with no cloud sync |
| Reminders | UserNotifications with calendar-based triggers and foreground/background handling |
| Accessibility | AVSpeechSynthesizer for spoken safety warnings; large-type controls |
| Document export | UIGraphicsPDFRenderer for PDF, manual CSV construction, system share sheet delivery |
| Embedded documents | WebKit for rendering bundled policy documents |
| Subscriptions | StoreKit auto-renewable subscriptions with entitlement verification |
| Calendar UI | Open-source calendar view component via Swift Package Manager |
| Networking | None — no HTTP client, no analytics, no crash reporting, no cloud services |
Technical Challenges & Solutions
| Challenge | Our Approach |
|---|---|
| Measuring a physical medical device to single-unit precision from a handheld camera | A deterministic pipeline that locates fixed physical landmarks by direct per-pixel classification and converts the measured pixel geometry through the device's known physical dimensions — explainable arithmetic rather than a probabilistic model, so a result can be reasoned about rather than merely trusted. |
| A transparent, curved, specular subject under uncontrolled household lighting | Ratio-based colour classification rather than absolute thresholds, so detection tolerates brightness variation; plus glare detection surfaced to the user as specific, actionable guidance rather than a generic failure. |
| Users with tremor, neuropathy or low vision holding the phone | Continuous gyroscope sampling at high frequency, with analysis suppressed entirely while the device is in motion — bad frames are never analysed rather than analysed and discarded, which also saves the processing they would have cost. |
| Any single frame can be blurred, glare-struck or mid-focus | Multi-frame consensus: results accumulate across sampled frames, outliers are trimmed against the running mean, and the most-repeated value is preferred once it is sufficiently dominant. Capture ends on genuine agreement, not on a timer. |
| Tuning a vision pipeline against real-world lighting rather than a lab | A parallel diagnostic pipeline that renders detected landmarks, scan geometry and measurement lines onto the analysed frame, making the algorithm's interpretation directly visible during development and field testing. |
| Full-resolution per-pixel analysis without stalling the camera preview | Cheap gates first — motion, then sharpness, then brightness — before the expensive scan; horizontal bounding of the scan region to the relevant band of the frame; background-queue dispatch with re-entrancy protection; and early termination once consensus is reached. |
| A safety-relevant number presented to a non-expert user | Layered interaction safeguards: an explicit confirmation window after which an unconfirmed reading expires, an always-available manual override with large discrete controls, and a spoken warning with a three-way choice when the measurement contradicts the prescribed schedule. |
| Sensitive health data with a trust-critical user relationship | A deliberately backend-free architecture. No account, no transmission, no third-party processor. Data leaves the device only through a user-initiated export, and the temporary export file is removed once sharing completes. |
| Reminders that stay useful rather than becoming noise | Notifications derived from the schedule and the reading history together and rebuilt on every change, so a reminder exists only where a dose is genuinely outstanding — rather than a fixed daily alarm the user learns to dismiss. |
Security & Reliability
No network surface. The application contains no HTTP client, no analytics SDK, no crash reporting SDK and no cloud storage integration. There is no server to breach, no credentials to leak, no transport to intercept and no third-party processor holding patient data.
Data stays on the device. Dose history, blood-sugar values, personal details and captured images are held in the local store within the application's own protected container, subject to the platform's file protection and the device passcode.
User-controlled export only. Health information leaves the device solely through an export the patient explicitly requests and explicitly shares, and the generated temporary file is deleted once the share sheet closes.
Explicit permission handling. Camera and photo-library access are requested with clear purpose strings, every authorisation state is handled individually, and denial produces guidance rather than a broken screen.
Confirmation before record. A measurement does not become a stored reading until the patient actively confirms it within a countdown window; unconfirmed readings expire.
Safety cross-check. Measured doses are compared against the patient's own prescribed schedule, with an audible warning and an explicit choice on any disagreement — a control on the application's most consequential output.
Human override as a first-class path. No measurement is final. The patient can always correct the value, and the correction path is designed to be usable by someone with impaired vision and reduced dexterity.
Graceful capture failure. Motion, blur, glare and lighting problems each produce distinct, actionable guidance, so a failed scan is recoverable by the user rather than a dead end.
Scalability & Performance
Tiered gating before expensive work. Motion state, sharpness and brightness are each checked before the per-pixel scan runs. The cheapest possible rejection happens first, so the costly path executes only on frames worth analysing.
Bounded scan region. Analysis is constrained horizontally to the relevant band of the frame rather than sweeping the full image, cutting per-frame work substantially without losing the landmarks that matter.
Background dispatch with re-entrancy control. Frame analysis runs on background queues with a guard preventing overlapping work and a short cooldown between passes, keeping the camera preview and the interface responsive while processing continues.
Convergence-based termination. Capture stops as soon as enough agreeing results exist rather than running for a fixed duration — faster results in good conditions, with a watchdog that halts scanning rather than letting it run indefinitely in bad ones.
Bounded reminder set. Forward-scheduled notifications are capped and rebuilt from current state on every change rather than accumulating, keeping the pending set small and always accurate.
No network latency by construction. Because nothing is fetched or uploaded, measurement, logging and review are immediate regardless of connectivity — which for a patient measuring a dose in a bathroom at six in the morning is the difference between a usable product and an unusable one.
Available performance headroom. A resolution-reduction step exists in the pipeline as a tuning lever, allowing per-frame cost to be traded against precision on lower-powered devices without restructuring the analysis.
Business Outcomes
- A dose can be verified without another person. The measurement that previously required good eyesight, a magnifier or a family member is performed by the phone and stated back to the patient.
- Wrong doses are challenged, not silently recorded. Because the application knows the prescribed schedule, a measurement that disagrees with it produces an audible warning and a deliberate choice rather than a quiet log entry.
- Adherence became evidenced rather than remembered. Every reading is timestamped and photographed, turning a recollection into a record.
- Clinical conversations have data behind them. A date-range CSV or PDF export gives a clinician the actual dosing history instead of a verbal summary.
- Reminders reflect reality. Because notifications are derived from what has and has not been logged, patients are prompted about doses genuinely outstanding rather than trained to dismiss a daily alarm.
- The product is usable by the people who need it most. Large-type controls, spoken warnings, motion tolerance and an always-available manual override target impaired vision and reduced dexterity directly.
- Privacy is architectural rather than promised. With no backend, no account and no third-party services, the claim that health data stays on the device is a property of the design rather than a policy statement.
- One device serves a household. Multiple local profiles let a caregiver manage more than one person without separate installations or separate devices.
Why it worked
Most computer vision work today begins by reaching for a model. Sometimes that is right. Here it was not — and knowing the difference is the point. The object being measured has fixed geometry and high-contrast landmarks, which makes the problem solvable deterministically: no training data to gather, no model to ship, no probabilistic output to explain to a regulator or a patient, and identical behaviour on every device. We chose the harder engineering path because it produced the more defensible product.
The rest of the work was the part that does not appear in a demo. Recognising that a single frame is never trustworthy and building consensus across many. Sampling the gyroscope so that shaking hands stop the analysis instead of corrupting it. Making the algorithm draw what it thinks it sees, so tuning against real household lighting was observation rather than guesswork. Distinguishing glare from misalignment so the user gets a fix rather than a failure.
And then the judgement that matters most in health software: deciding what the application does when it might be wrong. The measurement is checked against the patient's own prescription and warned about out loud. It has to be confirmed before it becomes a record. It can always be overridden. Every one of those decisions makes the software less confident and the product safer — and that trade is the one a healthcare engagement is actually asking you to get right.
Our team works across native iOS and Swift, camera and sensor-driven applications, on-device image processing and computer vision, offline-first architectures, accessibility engineering, and local data modelling with export and reporting — with the domain judgement to know when a problem should be solved with certainty rather than with probability.
Final Summary
Reading a dose off a small graduated medical device is a task that assumes good eyesight, steady hands and adequate light. The people who perform it most often have none of the three. The alternative — asking someone else — is not available three times a day, indefinitely.
Our team built a native iOS application that performs the measurement optically. The camera samples frames continuously; motion, blur and lighting problems are rejected before analysis begins; the physical landmarks on the device are located by direct pixel analysis and converted through known geometry into a dose figure; and agreement is built across many frames rather than trusted from one. The result is deterministic rather than model-based, which makes it explainable, identical on every device, and entirely offline.
Around the measurement sits the part that matters clinically. The application already knows what dose was prescribed for that day and time, so it checks the two against each other and speaks a warning when they disagree — while still letting the patient record what actually happened. Readings expire unless confirmed. Any value can be corrected through controls built for impaired vision. Each reading is stored with its photograph, reviewable in a calendar, and exportable as CSV or PDF for a clinical appointment. Reminders are derived from what has and has not been logged rather than fired blindly.
And none of it touches a network. There is no account, no server, no analytics and no third-party service. Health data stays on the device unless the patient chooses to export and share it — a property of the architecture rather than a promise in a policy document. What was an unaided visual judgement is now a verified, spoken, logged and evidenced one, performed on the phone the patient already owns.