Most LEDs need ~>3V (There are 5V LEDs as well, but not in the same diversity.) which means they need to be connected in series with a resistor, or they will be blown up by the current. The Teensy pins should be capable to provide ~40mA each (Probably not all at the same time..). Driving each row of LEDs one at a time, using one Teensy pin for each LED on that row should be possible, I think.
LEDs can be connected in matrices, much like switches are, and like supressing ghosting by pinpointing the exact switch being pressed, it is possible to control which exact LEDs to lit. I think it should be possible to use the same set of pins to read the switch matrix and control the LED matrix. I'm attaching a schematic for how I would do it. All component parameters still need to be chosen correctly. And this is of course under the usual caveat, I'm not an electrical engineer =)