When the mouse pointer is above a variable, the tool tip shows me the value of the variable.

But when the mouse point is above a constant defined with define('name','Value') then nothing is displayed.
And you can't see the defined constants in the variable section of the debugger as well.

It will be nice to be able to see the contents of the defined constants while debugging.

Yes!

I've requested this before somewhere as well. Definitely a wanted feature!

The only way to do it currently is to add the constant as a watched item.

you may want to use Watch or Immediate windows to check contants.
Alternatively you can just put get_defined_constants() function call there and see them all.