Enhance this!

Don't you hate when people have strong critiques towards a thing, but it happens that it's just that they don't know enough about it? Well, I don't, because then I think of how many times in my youth (and let's say only then) I did the same...

Regardless, today I happen to have a bit of time and I saw yet another post laughing at how stupid the "image enhance" trick used in movies and TV series is, and so you get this nerdrage against nerdrage...

Now think a second about this. Who do you think it's right? C.S.I., which is a huge TV series using arguably some of the best writers and consultants, or the random dude on the net? Do you think they don't know how realistic any of the techniques they use is?

Do you think they don't actually and very carefully thread between real science and fiction to deliver a mix that is comprehensible and entertains their audience, telling a story while keeping it grounded in actual techniques used in the field? Don't you think -they- know better, and the result was very consciously constructed? 
Well, ok sometimes producers just don't care, they want to tell a story, not write a documentary, but more often than not that's not the case.

The same goes of course for anything, really, especially when something is successful, makes a lot of money, has a lot of money behind, you should always bias yourself towards being humble and assuming the professionals making said thing -know better-.

Now, back to the "image enhance" trick. It turns out it is real science. It's called "super-resolution" and it's a deep field with a lot (really, a lot!) of research and techniques behind it.

It's actually common nowadays as well, chances are that if your TV has some sort of SD2HD conversion, well that is super-resolution in action (and even more surprising are all the techniques that can reconstruct depth from a single image, which also ship in many TVs, the kind of models they came up with for that are crazy!).

The scenarios presented in movies are actually -quite- realistic even if the details are fictionalized. True, the interface to these programs won't look like that, maybe they won't be real-time and surely they won't be able to "zoom" in "hundreds" of times, but they surely can help and surely are used. 

That is to me a reasonable compromise between fiction and reality, as certainly you can and will use computers to get a legible nameplate for a video that is too low-resolution for the naked eye, or match an otherwise unreadable face against a database of suspects and so on, probably not in quite as glamorous and simple way as the movies show, but fundamentally the idea is sound (and I'm quite sure, used in the real world).

It is a non-realistic representation of a very realistic scenario, which is the best that good fiction should try to achieve, going further is silly. Or are you going to argue that a movie is crap because at night for example you can't really see as clearly as they show, or because they don't let a DNA test take weeks and an investigation several years?

When it comes to videos we can use techniques known as "multiple image" super-resolution, registering (aligning) multiple images (frames in this case, i.e. optical flow), and merging the results, which do work quite well. Also, most fictionalized super-resolution enhances focus on faces or nameplates, which are both much easier to super-resolve because we can "hint" the algorithm with a statistical model (a-priori) which helps tremendously to guide the "hallucination".

And even if hallucinating detail might not hold in a court (the stronger the a-priori model, the more it will generate plausible results but by no means always reliable), it might be very well be used as a hint to direct the investigations (I've never seen a case where it was used in courts, always to try to identify a potential suspect or a nameplate, both cases where having a strong probability, even if it's far from certainty, are realistic).

So, bottom line is, if you think these guys are "stuuuuupid", well then you might want to think twice. Here are some random-ish links (starting points... google scholar for references and so on if you're interested... I couldn't even find many of my favorite ones right now) to the science of super-resolution:

• An example with nameplates, showing extreme reconstructions
• Another blind deconvolution paper, seems great on car plates
• Hallucinating faces
• An opensource implementation of video-based SR and a commercial multiple-image SR implementation
• Video to video SR, and another one...
• Classic paper of single-image super-resolution, trying to find similarities in the same image (to a degree, similar to fractal methods for image compression, which can indeed be used also for SR). Less strong results as the algorithm makes no prior assumptions.
• Multiple cameras (i.e. surveilance)
• More and more...
• An overview of commercial software used in forensics

It would take many pages only to survey the general ideas in the field. Don't limit your imagination... Computer science is more amazing than you might think... We reconstruct environments from multiple cameras, or even sweeping video... can capture light in flight, we can read somebody's heartbeat from video, fucking use lasers to see around corners and yes, even take some hints about an environment from corneas...

And by the way, don't bitch about Gravity, try enjoy the narrative instead. You might live a happier life :)