On Blocks, Stairs, and Beyond: Learning about the Significance of Representations