I just ran into a very frustrating bug in gluUnProject. It seems that if the model view matrix is rotated to align the view axis with the y-axis then gluUnProject is unreliable. Here's a little example:
Set up you model view, projection and viewport with:
MV = [1 0 0 0;0 0 -1 0;0 1 0 -4.3;0 0 0 1];
P = [1.81066 0 0 0;0 2.41421 0 0; 0 0 -1.00019 -0.0200019; 0 0 -1 0];
VP = [0 0 960 720];
Then given a point in the scene
p = [1.66667 0 0];
project it using gluProject (this seems to work OK):
proj_p = [816.867 360 0.99777];
move it one pixel up (as if with the mouse):
proj_p = [816.867 361 0.99777];
then unproject with gluUnProject:
wrong = [1.80114 -0.345327 -0.00534674];
Where did that movement in the y-direction come from? My guess is the implementation of gluUnProject is doing a bad job inverting the scene matrix in terms of floating point error. Luckily, libigl has an implementation of igl::unproject which uses Eigen's robust inverse() function. This solved my problem and gives the correct unprojection:
correct = [1.66667 0 -0.00494755];