Eugene Zozulya

OpenAI GPT is great for any task with plain text inputs. They are indeed pretty bad with structured knowledge like tables, knowledge graphs, database, etc. Their performance on 18 popular structure knowledge grounding tasks (like table QA, text-to-SQL, KBQA) are quite bad. They lag behind the existing specialized SoTA models by 20-50%. An interesting question we pursue is how to build foundation model to handle these structure knowledge. How far can we get with massive SFT on top of the open-source LLMs like LLaMA? Does scaling help with these tasks? We constructed a corpus containing 1.1M examples for both structured and unstructured knowledge grounding tasks to fine-tune LLMs. It turns out that such instruction tuning helps significantly. Our model achieves SoTA on seven different held-in tasks. The average performance on 18 held-in tasks is on par with all the specialized SoTA models. Our model also generalizes reasonably to held-out structure knowledge grounding tasks. Though being much better than existing models (UnifiedSKG and TableLLaMA), the gap with SoTA on these held-out tasks is still quite significant. Therefore, we tried to scale model from 7B to 34B. It turns out that scaling the model only yields marginal gains (3% boost). It seems that the scaling is not the no-brainer solution for these structure knowledge grounding tasks. We believe more work needs to be done to turn LLMs to structure knowledge experts.