iPhone X Depth Data Experiments

By January 20, 2018Uncategorized

Made some updates to this post to reflect the release of iPhone XS . . .

The iPhone features a front facing “TrueDepth” camera utilizing an inferredemitter that projects over 30,000 dots. An inferred sensor measures the time it takes the light to bounce back allowing an extrapolation of distance and therefor depth. This technique is called time of flight or ( TOF ) and it’s the same phenomena that allows for laser measurement to work, and bats to fly. iOS uses the data to accurately depth map your face, allowing for a secure and accurate method of facial detection. In addition iOS utilizes TrueDepth data in portrait mode to generate an accurate depth of field.

By contrast the rear facing camera currently does not use a TOF, instead it uses duel camera and another phenomena called the “parallax effect” to extrapolate the depth of field. Single lanes phones like the iPhone XS, go a step further by post processing the image using machine learning algorithms to determine a depth of field map simply from pixel data. So if you’re wondering why portrait mode selfies come out significantly better than the rear facing shots, it has allot to do with the TrueDepth camera data. If you have a single lens iPhone XR, you will also notice that the rear facing camera in portrait mode will only let you take portraits of faces with very shallow angles, by contrast the front facing and duel cameras can take photos of inanimate objects as well as people.

Since the release of iOS 12, not only do developers have access to ARKit, but we also no have access to pixel buffer data. This mean we can visualize, manipulate and uses the raw depth data. I ran a few experiments to visualize some of the available data.

I am using scene-kit to visualize the point cloud data that we extract from still portrait mode photos. Notice the subtle difference between the front facing (TrueDepth ) and rear facing camera. Surprisingly the true depth data seems to be less accurate than the camera data. This may have something to do with the way the point cloud data is being captured.

Here I are taking raw depth data from the video buffer and displaying it in a view as shades of grey. Raw depth data is a bit noisy, so you can see the black artifacts in the video. In this instance you can clearly see that the depth data output from the TrueDepth sensor is superior to that of the duel camera setup.

Finally we can use some of the inbuilt depth-kit functions to smooth out and filter noise. With the depth filter on, we can capture a fairly accurate depth map of the scene. Additionally can also apply a histogram to the output to calibrate our depth map.