Ben Hur retargeting challenge
Alec Jacobson
June 29, 2011
Recently I got the chance to see a friend's new image retargeting algorithm at work. It was very impressive and the best I've seen so far. I particularly liked that uniform scaling was within the set of possible solutions and actually occurred frequently. I like this because often uniform scaling looks way better than fancy retargeting techniques which sprinkle artifacts all over the image.
See the results reminded me again of Ben Hur. This movie was shot at a whopping 2.76:1 aspect ratio. Retargeting it to say, 4:3, would be a mess. All the times when I remember seeing it shown on TV (before widescreens became popular) it was shown with "letter boxes" rather than "pan and scan" or any other attempt to retarget the footage. Not even considering the fact that the original aspect ratio was intentional and that by retargeting it you'd be trivializing the directors' choices (would a museum crop or stretch a painting to fit the gallery walls better?). Not even considering this, Ben Hur poses a striking challenge for retargeting techniques. To me it represents a real stress test. Much harder than most of the images in the RetargetMe dataset.
Here are some stills I found flipping through the film that I thought would be particularly difficult to retarget:
I ran some 4 of the easiest methods for retargeting on the last image.
Letter boxing
Uniform scale
1/2 Uniform scale, 1/2 letter box
Seam carving
Personally none of the above options besides Letter boxing are acceptable. Not even close. I would really like to see a retargeting algorithm that not only includes uniform scaling and cropping in the set of possible solutions, but also letter boxing. Do you know of one?
It goes without saying that the problem is much harder when considering not just a single frame but the entire scene or film's worth of video. Letter boxing may be harder to include as an option when considering temporal coherence, but I still think it must be a possible solution. For films like Ben Hur, which purposefully utilize their entire aspect ratio, squishing a scene or cropping out elements would be far worse than the price of two black bars.
I ran some 4 of the easiest methods for retargeting on the last image.
Personally none of the above options besides Letter boxing are acceptable. Not even close. I would really like to see a retargeting algorithm that not only includes uniform scaling and cropping in the set of possible solutions, but also letter boxing. Do you know of one?
It goes without saying that the problem is much harder when considering not just a single frame but the entire scene or film's worth of video. Letter boxing may be harder to include as an option when considering temporal coherence, but I still think it must be a possible solution. For films like Ben Hur, which purposefully utilize their entire aspect ratio, squishing a scene or cropping out elements would be far worse than the price of two black bars.