A key thing limiting reasoning about 3D human-object interaction in images is the lack of training data. In PICO, we propose a novel way to label human-object contacts in images and build a new dataset with dense, vertex-level, contacts on the humans and objects. We then infer the 3D object shape and pose, human shape and pose, and use contacts in an optimization method to solve for a consistent 3D configuration.