The Russo brothers actually answered this exact question on the Infinity War commentary track and it wasn't the Soul Stone. In the first Avengers movie, Thanos gave Loki an army and the Mind Stone to conquer Earth. Everything was going according to plan until one guy in a red and gold suit carried a nuclear missile through a portal into space and wiped out Thanos's entire fleet. Because of Tony’s play in New York, Thanos lost both the Tesseract (Space Stone) and the Scepter (Mind Stone) in a single day. You don't let a mortal human blow up your flagship and steal two Infinity Stones without immediately pulling up his file to see who he is. Thanos spent six years watching Tony Stark build suits out of paranoia for his arrival, which is why he tells him, "You're not the only one cursed with knowledge". #Marvel #MCU