Planets are objects better observed in the infrared. Using CRIRES as "role-model" of a high-resolution infrared spectrograph various issues in the context of detection and characterization of brown dwarfs and extra-solar planets or even comets will be reviewed. The stability limits of CRIRES "as-is" will be reported. The potential of anticipated upgrades such as an image slicer will be discussed. While CRIRES at the 8m telescope will at best provide for orienting measurements, a dedicated instrument at an ELT could have an impact thanks to the quantum leap in sensitivity.