Kinect Hacking 101: Hack a Powershot A540 for Infrared Sensitivity.

Microsoft’s new Kinect sensor is garnering a lot of attention from the hacking community, but the technical specifics of how it works still aren’t clear. I am working to understand the technology at a fundamental level – my interest is in the optical side of Kinect. My ultimate goal is to make the sensor nearsighted, so that the depth resolution can be used to scan small objects. The first step in understanding a technology is to look at it — that’s why teardowns like this one at iFixit are so important.

Unfortunately for us humans the Kinect projector is infrared (or near-infrared) — probably somewhere between 900 and 1020nm. My guess is that it’s around 904nm, because it’s cheap to produce those lasers, but that’s just a guess; I haven’t measured anything yet. That means that by design, we can’t see the projection directly with our eyes. But we can see it with a camera. Almost all CMOS sensors in digital cameras ARE sensitive to infrared. In fact, they are so sensitive that there is a filter placed between the lens and the sensor to prevent IR from messing up your pictures. It’s called the IR cutoff filter.

The IR cutoff filter is easily identifiable. It’s a bluish piece of glass. It’s blue because the glass absorbs wavelengths that are just longer than red. This is an IR cutoff filter from an old video camera, you can clearly see how the filter just sits on top of the sensor:

Here’s the IR cutoff filter in my Sony NEX-5:

These filters are so effective in reflecting and/or absorbing IR that the reflected IR has been used to detect cameras in dark theaters! Yet another effective anti-piracy strategy… I see it’s been steadily declining since that was published in 2004.

Anyway, we need an IR sensitive camera to see what the hell is going on. I’d rather not use a cheap IR security cam — the resolution is just too low. And I’m not going to modify my NEX-5 for infrared (yet – but see Pete Ganzel’s excellent work on that!). So I’ll have to compromise and pick what is probably the most difficult option: modifying a compact Powershot camera. It’s difficult, because the filter in a compact camera is INSIDE a closed lens mechanism, and the camera was never designed to be disassembled. This tutorial will be a little incomplete, but should be enough to get any motivated person done with the same project. As all hackers know, it only takes one person to get the work done and share the results.

Start with a clean workspace. Like this one.

Disassemble the camera screw by screw. Don’t miss the hidden one inside the battery compartment.

Carefully pull the body apart:

The back will come off first. It may be blurry. Don’t let that bother you:

Next, the front:

Remove the display by removing the screw at the top left and unclipping these metal clips:

Next, carefully unclip the FFC (Flat Flexible Cables) from their mounts. Some will be pull-out, and some will have a brown plastic clip that must be flipped up to release the cable. The display will come free, but it will be embarrassed and hide from the camera.

Remove the keypad assembly, and unscrew all screws holding in the mainboard. Unfortunately, for the Powershot A540, I also had to unsolder this red and black wire which supply power to the flash. Since I hate the flash, and the camera is not dependent on it, I simply removed it for good. An IR cam has little use for a flash. Good riddance.

Under the mainboard, you can see the back of the sensor. There is a bit of hot glue on a metal plate which is supported by three screws. This is the sensor assembly; it is our target. Now, before I go much further, I need to say that the camera will be difficult to return to normal operation. The IR filter must be destroyed to get it out. This forever changes the focal characteristics of the camera unless it is replaced with flat glass (and I’m not doing that). So adjust your expectations of how perfectly the camera will focus from here on out, and all will be well.
Remove the mainboard.

Here is a better shot of the sensor assembly. The three screws are covered in hot glue. That is because next to each screw, there is a small spring which pushes up on the sensor assembly. In other words, it’s a little 3-way platform that the sensor is adjusted on. The exact position is fixed because it was calibrated at the factory for perfect focus. We are going to destroy that by removing all three screws very slowly. Slowly, because the three springs will pop out if we move too quickly.

Remove the three screws. Lift the sensor assembly free. Springs are visible.

Well, there it is. The IR cut filter, held in by two dabs of hot glue. Try to lift it with a small screwdriver or hobby knife. It will most likely break because it is very thin. Remove the little pieces. Admire their lovely blue hue.

Now comes the really hard part — you have to put the camera back together. I did so without the flash assembly in place, and I left a few screws out. My Dad taught me how to take things apart, and how to re-assemble them. He always called these “extra” screws “shipping screws”. Here are the shipping screws left out of this camera:

Alright! I’ve completed the very first tool in my Kinect hacking suite, a camera to look at the Kinect’s pattern. But the question is: Was all this worth it? Does this really tell us anything? See for yourself:

Before the infrared modification:

After the infrared modification:

Alright, looking good! Let’s make a movie:

This is kickass! Here’s what the sensor output looks like up close — this board is about .5M away from the camera (click for full resolution, slightly misfocused – this image is public domain – I make no claim to copy rights):
null

It’s interesting in a few different ways. One can easily see that the Kinect IR speckle-field is a 3×3 matrix of random dots. One can also see that they are differentiated by intensity and have a centered registration dot. So it is not purely random speckle.

IR cameras are good for more than Kinect hacking. Here are a few outdoor shots showing some IR effects.

Stay tuned as I build up a proper toolset for optical analysis and decipher just how these guys are using speckle to estimate depth.

This entry was posted in ComputationalPhotography, Photography. Bookmark the permalink.

16 Responses to Kinect Hacking 101: Hack a Powershot A540 for Infrared Sensitivity.

  1. Maradatscha says:

    Do you know any way to grab the raw output of the 3 kinect cameras, instead of 1 rgb and 1 depth stream?

  2. admin says:

    There are only two cameras and one projector. It may be possible to get raw frames from the depth camera instead of depth information. My guess is that if it were possible, it would happen during the calibration routine with the xbox. As we get further along in hacking, such things may become common or easy but they aren’t just yet.

    I wish someone would do a USB dump on the calibration routine.

  3. Dustin Kerber says:

    Ladyada would more than likely be happy do a dump on the calibration if someone asks.

  4. admin says:

    I’ll ask, but I’ll leave the approval up to her. :)

  5. rikarends says:

    Try reading the patent applications for the thing,
    it will probably help you along quite far (if not all the way).
    Its not a how-to but it clearly says how the thing works such as
    how the dots are randomized, why, etc.

    http://www.faqs.org/patents/app/20100118123
    http://www.faqs.org/patents/app/20100020078
    http://www.faqs.org/patents/app/20100007717

  6. admin says:

    Yeah, I have those, and half a post written on them… Thanks for the links though! It’s absolutely the right track.

  7. draeath says:

    It would be interesting if you had a glass suitable to do the reverse of the IR filter – filter anything above near-visible infrared!

  8. admin says:

    Actually, I do have that. It’s going to be a part of my next post. In short, the way to do it is to use two crossed polarizers. Two crossed polarizers pass only infrared light. A cheap, simple optical filter!

  9. Dean Du Bois says:

    Nice image of the IR projector output. I think you will find that the dot pattern encodes the projected XY location (along the lines of Anoto http://www.anoto.com/?id=906). Three or four reflected points could be used to determine the depth and orientation. The dot patterns would be converted to 11 bit depth information by the PrimeSense chip). a fish-eye lens on the IR camera may return data you could recalibrate for higher resolution short range measurements.

  10. admin says:

    It’s a little more complicated than that, the speckle pattern is more or less depth-invariant. The patents and papers say that it’s actually a decorrelation between known patterns calibrated at different planes that give the depth data. A fisheye may work, and it’s one approach that I’m investigating, but the issue is that any glass in front of the projector changes the speckle pattern, and so requires recalibration. It’s going to be a very interesting hack session, and I look forward to really deeply understanding the specifics of how this device is working.

  11. barry99705 says:

    Hey, just to let you know. A piece of exposed and developed 35mm film makes a pretty good ir filter as well.

  12. wulfderay says:

    A piece of the medium from a 3.5 inch floppy disk will also work as an ir pass filter.

  13. Pingback: SpaceTechsOnline.net » Blog Archive » How-To: Modded camera looks at Kinect infrared output

  14. Pingback: How-To: Modded camera looks at Kinect infrared output | ignacio19hensley

  15. Henk Poley says:

    Next time use an icecube tray to lay out the screws in the order you took them out. Makes it much easier to put it back together correctly.

  16. two quick ideas for scanning things nearer the kinect:

    1 perhaps it’s an intensity issue. in this case, place a neutral density filter over the projector or camera.

    2 perhaps it’s a displacement issue — i.e., the pattern just isn’t visible to the camera. in this case, a reverse fisheye won’t work. instead you can just use two mirrors like a periscope to move the projector closer to the camera.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>