Made some updates to this post to reflect the release of iPhone XS . . .

The iPhone features a front facing “TrueDepth” camera utilizing an inferredemitter that projects over 30,000 dots. An inferred sensor measures the time it takes the light to bounce back allowing an extrapolation of distance and therefor depth. This technique is called time of flight or ( TOF ) and it’s the same phenomena that allows for laser measurement to work, and bats to fly. iOS uses the data to accurately depth map your face, allowing for a secure and accurate method of facial detection. In addition iOS utilizes TrueDepth data in portrait mode to generate an accurate depth of field.

By contrast the rear facing camera currently does not use a TOF, instead it uses duel camera and another phenomena called the “parallax effect” to extrapolate the depth of field. Single lanes phones like the iPhone XS, go a step further by post processing the image using machine learning algorithms to determine a depth of field map simply from pixel data. So if you’re wondering why portrait mode selfies come out significantly better than the rear facing shots, it has allot to do with the TrueDepth camera data. If you have a single lens iPhone XR, you will also notice that the rear facing camera in portrait mode will only let you take portraits of faces with very shallow angles, by contrast the front facing and duel cameras can take photos of inanimate objects as well as people.

Since the release of iOS 12, not only do developers have access to ARKit, but we also no have access to pixel buffer data. This mean we can visualize, manipulate and uses the raw depth data. I ran a few experiments to visualize some of the available data.

I am using scene-kit to visualize the point cloud data that we extract from still portrait mode photos. Notice the subtle difference between the front facing (TrueDepth ) and rear facing camera. Surprisingly the true depth data seems to be less accurate than the camera data. This may have something to do with the way the point cloud data is being captured.

Here I are taking raw depth data from the video buffer and displaying it in a view as shades of grey. Raw depth data is a bit noisy, so you can see the black artifacts in the video. In this instance you can clearly see that the depth data output from the TrueDepth sensor is superior to that of the duel camera setup.

Finally we can use some of the inbuilt depth-kit functions to smooth out and filter noise. With the depth filter on, we can capture a fairly accurate depth map of the scene. Additionally can also apply a histogram to the output to calibrate our depth map.

ARKit

At WWDC2017 Apple announced to developers they would be able to access features of the new ARKit framework. The framework is designed to facilitate the building of Augmented Reality experiences on iOS devices. It utilizes Apples A9 – A10 chips and will presumably work with increasing sophisticated sensors that are being developed for upcoming iPhone releases.

Digging into some of the documentation around the beta version of ARKit, apple provides documentation on how to place objects with an 3D augmented reality context.

Once we place those objects or planes we can get the x,y,z coordinates of each node we created. By then doing a little bit of vector math, we can compute the distance of the two points on a two demential plane.

The Demo

A video of the demo we built is featured here, ARKit is calibrated in meters so we need to convert that to inches. According to the app, the long side of the rug measured 1.95 meters or 76.77 inches. The actual distance as measured by the tape measure is 77 inches. Being off by less than 1/4 of an inch is shockingly accurate and could potentially be attributed to where I dropped the circles in the scene.

Looking forward to testing out additional ARKit functionality as it evolves.