SeeSpace InAir promises a whole new way of interacting with your television. Using an inline HDMI dongle, they hope to enhance your favorite programming with additional information from the web. It’s definitely a cool idea if done well, and the hardware technology is there, but the real question is where this extra content is going to come from.
The InAir’s hardware is pretty straight forward. This little dongle sits between your cable box and TV, and when powered by a USB cable, it will overlay all kinds of cool information on top of your regularly scheduled programming. Google’s Chromecast offers HD internet streaming to a small dongle and retails for a piddling $35; certainly there are a number of solutions that SeeSpace could be using to make this device work.
A careful perusal of their campaign however showed no mention of HDCP. HDCP or “High-bandwidth Digital Content Protection” is a form of hardware DRM that is built into a number of television screens and digital media players. It’s designed to encrypt the data leaving a media player to prevent hardware from recording it and making unauthorized copies. The only way to make sense of this data is to license the HDCP technology that allows you to decrypt this signal. Most major television manufacturers have done this as there is no way to digitally record information presented on a television screen, but the more a device looks like a video recorder, the harder it is to get certified.
In a comment, they stated that it will be HDCP compliant:
But they haven’t gotten certified yet. A whitepaper on the topic had this to say about compliance requirements:
Licensed technology Adopters are required to meet content protection requirements. For example, the HDCP License Agreement prohibits high-definition digital video sources from transmitting protected content to non-HDCP enabled receivers, and such devices from making copies of decrypted content.
If their content recognition algorithm requires any video frames to be transmitted over the internet and stored, they may not qualify for HDCP compliance. What this means for the consumer is that certain cable boxes that require HDCP will not work with the InAir. If your cable box (or any future cable box) requires HDCP, I would ask SeeSpace to confirm their certification before backing.
The real issues are with the software. The features demonstrated show a very cool TV viewing experience. Take for example this race overlay:
Sure, baseball and football broadcasts will contain a little counter in the corner that gives you all that you need to know, but there’s a lot more data to display for a race. This system allows the user to have access to a large amount of information as it unfolds.
The big question is who is providing this information? The InAir description mentions that this display is a “port” of the existing Formula 1 app, so the Formula 1 app people are supplying it in this case. This app is supported by the F1 organization, so they can stream information from a race that may not make it to the TV broadcast.
What SeeSpace doesn’t mention is that the app costs $14 and that it is currently only available for a few mobile devices. My first thought is that they might be running the app in some sort of heavily modified Android OS, but the integration is just too good for that to be the case. For example, the app does allow a 3D view of the course, but it looks more like a Google maps satellite view than the sleek 3D map shown. The map which precisely matches the theme of the InAir’s user interface.
In order to produce a screen like this, SeeSpace would need to have their own custom application that interacts with the Formula 1 app’s backend. Building this would require access to the app’s source code or at least some kind of API which would require some kind of partnering with the developers as it is not an open-source app. After contacting the F1 app people regarding such a relationship, I got this response:
No we have not provided them anything. It looks like they have created
a mash up of various things including our assets. Whilst this is not
technically legal in the UK, its a mash up, so probably fair use in US
The biggest challenge will be commercial viability for a very small platform.
This directly contradicts what SeeSpace has said:
Strange that they could port it without the source code or anything else from the F1 app team. There’s a very good possibility that they are lying about porting this app. It’s likely a simulation of what the final product could do, not a demonstration of what it does.
What about television content that doesn’t even have an associated app? They demonstrate a rather elaborate set of features for enhancing a documentary on space exploration:
The video in this case appears to be from the BBC documentary series “The Planets” which was released in 1999. A standard-definition version is actually available on Netflix if you’re interested:
I just finished watching it myself, and it was excellent.
The video demonstrated on the InAir overlays this documentary with a 3D model of the Voyager probe that it supposedly found on the internet all by itself. I found that a search of “Voyager probe 3d model” brought me to a NASA page which contained several 3D models of space probes. I found two different models of Voyager: one in .3DS format and one in .blend:
Both of these models required special software to view.
It looks like they’re using the Blender model in the video (based on some of the detail and texturing not present on the 3DS model). This model looks great, but it lacks the labels shown in the video. For that level of information, I needed to find a diagram such as this one on Wikipedia:
Which seems to agree more or less with the labeling the InAir showed.
So what you have here is a sort of “book report” being assembled automatically about the content shown on the screen. This kind of information is very useful for someone wanting to learn more about space exploration, and it would certainly enhance any viewing experience.
The only problem is there’s very little explanation for how the InAir is doing it. They offer the following:
This example shows the possibilities of projecting 3D models into the room to navigate and explore. It also demonstrates the way InAiR can access open web information such as Wikipedia and other relevant sites on the Internet. In this example it’s the NASA website and its library of photographs.
Okay, so let’s talk about the number of things that have to go right for this particular overlay to work:
- The software needs to identify that the documentary is currently talking about the Voyager probe. According to their FAQ, they’re using “channel guide, tele-text, and voice layer to understand the context of what you’re watching based on keywords.” We all know how reliable that can be. I’m not certain that the TV guide data would even contain the word “Voyager”, so they would need to pick it (and its context) out of the subtitled broadcast with the program (if available) or use voice-recognition which is extremely challenging if there is any background noise/music coming from the source.
- To find information on this space probe, the InAir would need to search the internet and somehow know to filter out any results related to the Star Trek television series of the same name.
- Assuming the InAir somehow understood automatically that the Voyager probe is a physical object, it would need to search for a 3D model and download it. There is more than one model available, so it would have to arbitrarily pick one of them. The file is zipped, so after unzipping, it would need to identify the Voyager .blend file from the PNG texture files and find a suitable way to open it. There are dozens of different 3D model formats and even more still image formats. It would need a port of a Blender and 3DS Max viewer. It would also need to figure out how to orient the file. The .blend file had the probe tilted 90 degrees from how it was shown in their demo.
- After opening the file, it would need to identify the parts of the probe on that 3D model and cross reference it with a number 2D diagrams found elsewhere on the internet. This is an incredibly difficult machine vision problem to solve especially considering that many of the 2D diagrams online are line drawings that bear little resemblance to the 3D model (from a computer’s point of view).
- Unless you want to see a spinning “loading wheel” all of this needs to happen nearly instantaneously. The device will also have no time to prepare if you flip to a particular channel and immediately start looking for augmented content.
In short, there is a lot of difficult work that this machine has to do to present the information shown. Google does this kind of work with some of their more general search results:
This is the kind of information that InAir is trying to show, but even Google, the multi-billion dollar company whose mission statement is “to organize the world’s information and make it universally accessible and useful.”, can only display a few images, a summary ripped from Wikipedia, and a few dates that are probably also ripped from Wikipedia. The level of detail shown in the automated results on the InAir overlay is eons beyond what even Google has accomplished. If they can really accomplish something this sophisticated, they shouldn’t be selling a TV dongle. They should be selling this algorithm to universities.
The only way I can see something like this being even remotely plausible is if a human or team of humans is somehow involved in curating the content displayed; there is no way a computer today can automatically provide the information shown while also filtering out all of the other noise on the internet.
Perhaps the content providers will do it themselves? Their FAQ states:
What I can tell you is that content providers are interested, but I can’t talk specifically about their interests.
Uh huh… I’m more inclined to believe the F1 guys. The platform is too small to be profitable.
Can they do it?
When you realize how many television programs there are out there, you might start to realize why this is such a staggeringly difficult problem to solve unless it’s totally automated. SeeSpace is sweeping all of those important issues under the carpet and focusing instead on gee-whizzing people with a fancy user interface. Even in their “Risks and Challenges” section, they talk entirely about manufacturing concerns without mentioning this back-end at all.
The only time they address the algorithmic side of things is here:
We have a patented InAiR content recognition engine, based on our own algorithms, which intelligently identifies relevant Internet and social content with what the viewer is watching and delivers it to the TV screen in real time.
I found a few relevant patents from Dale Herigstad involving video delivery, but none of them mentioned any special algorithms for content recognition or data acquisition. I left a question in their user comments section asking for a patent number, but they appear to be ignoring me.
They’re also responding to legitimate backer technical questions with an unsettlingly coy attitude. When one backer was discouraged by the lack of Video on Demand support (Netflix, etc), SeeSpace responded with:
They never mentioned Video on Demand in their project description; it only comes up in their FAQ which isn’t even linked from the Kickstarter page. It’s not a dumb question, but they’re treating it like it is.
Their vague reply and disparate information almost makes it sound like their product spec hasn’t really been nailed down yet. Makes one wonder how much they’re comfortable changing when it comes to actually shipping. If Netflix is a major feature, advertise it and stand by it. Don’t make off-hand remarks in your comment section.
Without more information on how the automation system works, I have to conclude that this device simply will not function as advertised and that the demonstrations shown so far are forgeries created with specially hand-curated content that is not attainable with an automated system or sustainable with a human one. Sure, there may be certain programs like Formula 1 racing that work (assuming they actually do some day get access to the app’s source code), but for every-day use? Forget about it.
The Voyager probe had some of the most sophisticated data collection equipment available at the time, but it still needed humans to interpret what it found.