Generating synthetic data that maintains realistic relationships between columns is crucial for testing and analysis. Traditional random generation approaches often create unrealistic patterns, like luxury hotel rooms priced cheaper than basic rooms. GaussianCopulaSynthesizer automatically learns and maintains these relationships, creating synthetic data that preserves the statistical patterns of your original dataset. ⭐️ Full code: datacebo.com/dev-posts/sdv.h…

Many businesses collect and store their customers’ GPS locations to help improve their products. But GPS locations may contain precise locations of people’s homes. Businesses are sensitive to sharing this data even to internal teams, as it may reveal private information about people they know. For example, a food delivery application stores the GPS location associated with each delivery. An internal product team wants to use this data to improve the local restaurant recommendations the application makes to users for future orders. The company needs a way to preserve local insights on the best restaurants from the GPS location data without exposing sensitive user locations. One anonymization approach they could take is replacing every collected GPS location with a randomly chosen one from within the same postal code. Users tend to order from restaurants in the same or neighboring postal codes, so the integrity of local trends is still preserved. To implement this approach, they would need a dataset that contains the geographic boundaries for each postal code and an algorithm for identifying the postal code from a GPS location. To make this process seamless, we created the MetroAreaAnonymizer. With just a few lines of code, you can use the MetroAreaAnonymizer to replace GPS locations with a randomly chosen one from the same postal code. MetroAreaAnonymizer is part of our RDT library, which contains many helpful transformations for your raw data. 📚 Learn more about MetroAreaAnonymizer here: docs.sdv.dev/rdt/transformer… 📚 Learn about RDT here: github.com/sdv-dev/RDT 📚 Learn more about the SDV here: sdv.dev/ #syntheticdata #machinelearning #anonymization #geospatial

Synthetic tabular data can help you test software applications because it resembles the key properties and patterns in your real data. Consider a news publication that wants to use synthetic data to test a new software change for their mobile application before it rolls out to their entire reader base. They trained an AI model on their real data and used it to generate synthetic data. Before they can incorporate this synthetic data into the test environment however, it must meet some minimum criteria for the application to function properly. Here are some examples of criteria that the synthetic data must meet: 1. Data Validity: Primary keys must be unique and non-null. Many features need to retrieve a specific row in a table using a unique identifier. For example, to authenticate a user, the application needs to find the specific row corresponding to their unique user_id value. 2. Data Structure: Data types, column names, and table names should match those in the real data. Application code that retrieves or updates data using specific column names, column types, and table names will error, like when the application needs to update a user’s settings. 3. Relationship Validity: Each foreign key must have a reference to a valid primary key (also known as referential integrity). Many features in the app require joining data from multiple tables, like the recommended articles feature. Without referential integrity, the retrieved data might contain a subset or none of the recommended articles for the user. To help them validate that the synthetic data meets the minimum criteria for usability, they could use the SDV’s Diagnostic Report. This report runs all of our basic data format and validity checks by comparing the real and synthetic data. The Diagnostic Report is part of our open-source and vendor-neutral SDMetrics library. Synthetic data generated by the default synthesizers in the SDV will always result in 100% diagnostic scores. We call this the 𝗦𝗗𝗩 𝗚𝘂𝗮𝗿𝗮𝗻𝘁𝗲𝗲. If the SDV ever generates synthetic data that doesn’t score 100% on the Diagnostic Report, then you’ve identified a bug! Please reach out to us on GitHub or Slack and we will prioritize investigating it. 📚 Learn more about the single-table Diagnostic Report: docs.sdv.dev/sdmetrics/repor… 📚 Learn more about the multi-table Diagnostic Report: docs.sdv.dev/sdmetrics/repor… 📚 Learn more about the SDV here: sdv.dev/ #dataquality #generativeai #machinelearning #softwaretesting #syntheticdata

One challenge in training AI models to generate valid synthetic data is teaching them to mimic the rules-based business logic that exist in real datasets. Let’s explore an example of one such rule. The one-to-many relationship is a common pattern in database schemas. An interesting variation of this pattern occurs when only some rows are allowed to have connections while others aren’t. For example, a gym offers a premium membership tier that gives access to extra benefits (like pool access and sauna access). To record the perks available to each member, they use a members table and a benefits table. Only the rows representing premium members are allowed to have connections to rows in the benefits table while the rows representing basic members are not. This enables the gym to store specific information for a subset of their membership in a separate table in a simple way. We call this the ForeignToPrimaryKeySubset pattern because only a subset of the primary keys in the parent table have a 1-to-many relationship with the foreign keys in the child table. If your data contains this pattern, you can now generate multi-table synthetic data using the SDV that also adheres to this pattern. This pattern is part of our Constraint Augmented Generation bundle, or CAG, in the SDV Enterprise. 📚 Learn more about the ForeignToPrimaryKeySubset pattern here: docs.sdv.dev/sdv/reference/c… 📚 Learn more about the CAG bundle here: docs.sdv.dev/sdv/reference/e… 📚 Learn more about the SDV here: sdv.dev/ #syntheticdata #generativeai #databases #machinelearning #datamodeling

✈️ @Expedia recently shared a very interesting methodology on how they collect and use synthetic data to improve their flight price forecasting models. When a user makes a flight search, Expedia retrieves the latest pricing data from their data providers for the specified search parameters - route, fare class, trip dates, etc. To build interesting price prediction features for their customers, the Expedia team trains forecasting models on data they’ve collected but they wanted to improve prediction accuracy even further. 🛑 The Challenge Even though millions of searches are made by users daily, the sheer number of combinations for possible routes, trip dates, and passenger counts is so large, that there were a lot of combinations for which the team did not have the price. To develop a robust forecasting model ideally the team would have at least one search a day for each of the combinations of the search parameters. 🤖 How they Incorporated Synthetic Data? To fill these gaps they built automated software that requests flight prices for specific search parameters. 🎯 Their goal with synthetic searches is to have at least one search a day for their most popular routes for the trip dates that fall within the upcoming months. During the model training phase, they combine data from real user searches and from synthetic searches to ensure they have better data coverage. ✅ User Impact When a user searches for a flight, Expedia shows a chart that visualizes how prices are forecasted to change between now and takeoff. By improving the accuracy of their price forecasts, Expedia helps their users decide if they should book a flight immediately or wait until a forecasted price drop occurs in the future. 🚧 Limitations Using an automated search based on synthetically created search parameters could interfere with the experience of onsite users - who are trying to search for price. The team took this into consideration and were deliberate about balancing the data retrieval needs of real user searches with the team’s needs for synthetic searches. 📚 Read the Dec 2024 @thenewstack article by Shiyi Pickrell, the SVP of Data and AI at Expedia: thenewstack.io/the-future-of… 📚 Read the Oct 2023 @Medium article b y Andrew Reuben: Senior Machine Learning Scientist at Expedia: medium.com/expedia-group-tec… #syntheticdata #generativeai #machinelearning #openai #travel Image credit: Expedia

One challenge in training AI models to generate valid synthetic data is teaching them to mimic the rules-based business logic that exist in real datasets. Let’s explore an example of one such rule. Some applications need to store numerical data with different units of measurement in the same column. For example, an online retailer accepts payments in many different currencies and records every transaction in a table. They use an amount column to record the transaction amount and a currency column to record the currency for each transaction. The transaction amounts associated with each currency might have radically different scales (min-max ranges and distributions) because of the exchange rate. 1 USD (American Dollar) is equivalent to ~1063 ARS (Argentinian Pesos), which is reflected in the transaction amounts. We need a way to instruct the AI model to learn the scales for each currency separately. To enable SDV synthesizers to model this business logic and generate synthetic data that adheres to it, we created the MixedScales constraint. You can use this constraint whenever the value of one or more categorical columns (like the currency column) determines the scale of a numerical column (like the amount column). The MixedScales constraint is part of our Constraint Augmented Generation, or CAG, in the SDV Enterprise. 📚 Learn more about the MixedScales constraint here: docs.sdv.dev/sdv/reference/c… 📚 Learn more about the CAG bundle here: docs.sdv.dev/sdv/reference/e… #syntheticdata #generativeai #databases #finance #datamodeling

Today, we’re excited to introduce a powerful new bundle to The Synthetic Data Vault: AI connectors. AI connectors address 2 key challenges that SDV users face when training generative AI models on datasets from enterprise data stores. (Link to the announcement: bit.ly/3EURLCB) ❎ Creating accurate metadata is time consuming, especially for complex multi-table schemas Metadata provides a deeper context (semantic and statistical) about your data and the synthesizers use this context to generate high quality synthetic data. Without AI connectors, SDV users have to export data from the database, use SDV’s metadata auto-detection feature to establish metadata, and then manually update the metadata to be accurate. ✅ AI Connectors automatically generate higher quality metadata AI connectors automatically infers higher quality metadata using the database schema and our own inference engine, without having to read tables into memory from the database. When benchmarked with 55 datasets stored in 4 different database platforms, metadata generated using AI connectors resulted in 35% higher quality metadata (average score of 0.98) compared to metadata generated using the auto-detection approach (average score of 0.73). ❎ Identifying a referentially sound and representative sample for training data is tricky Training SDV Synthesizers requires loading a representative sample of data from your database into memory. In addition, the data needs to have referential integrity for the synthesizers to learn the proper relationships. Approaches to identifying a high quality, referentially sound sample of data can be tedious and time-consuming to implement. ✅ AI Connectors uses an inbuilt algorithm to generate a training data set and guarantee referential integrity With AI connectors, we created an algorithm called Referential First Search (RFS) that guarantees that the real data used to train the model is a subset with referential integrity. When benchmarked with 7 datasets stored in 5 different databases, training data created using AI connectors achieved an average of 18% higher quality data score over the standard approach of random subsampling and then enforcing referential integrity after. Read more about AI connectors and how to access it in our latest product announcement here: bit.ly/3EURLCB #syntheticdata #generativeai #machinelearning #databases

SDV Enterprise v0.23.0 is out 🎉 This release enhances your ability to program your synthesizer to find certain patterns and recreate them— whether it's through multi-table CAG patterns, single-table constraints, or pre-processing techniques that transform your data. 🏆 Improved CAG patterns. Use CarryOverColumns to specify a column that is repeated across many tables with different relationships. The PrimaryToPrimaryKeySubset pattern now works with missing values. See more about these interesting data patterns SDV Enterprise supports in the slides below. 💡 Experiment with new transformers to improve your synthetic data quality. Try applying the new LogScaler and LogitScaler on data that exhibits exponential properties. 📚 Read the full Release Notes here: bit.ly/4152LVn 📚 Learn more about the SDV: bit.ly/4b858Lu #syntheticdata #generativeai #machinelearning #ai

Synthetic data is a powerful way to generate test data that looks and feels like real production data. You can either insert the synthetic data back into the database in an environment for manual testing or use the data for running automated tests. But if you need to test a new application that has no real world usage or collected data, then you need to adopt a different approach. Instead of training models on your real data to generate synthetic data, you can generate fake test data from scratch that adheres to your database schema. In the SDV, we created a dedicated synthesizer called DayZSynthesizer to support this workflow. Here are the 3 main steps: 1. Generate baseline metadata Auto-generate baseline metadata from your database’s schema (for supported databases) or use our Metadata APIs to create a JSON representation of your metadata that mirrors your database schema. 2. Improve the data realism You can update sdtypes to add semantic meaning to special columns like social security numbers, postal codes, and addresses to improve the format and type of fake data that’s generated. You can also define min-max value ranges for numerical columns, define a fixed set of categories for categorical columns, define datetime ranges, and control the proportion of missing data you’d like for each column. 3. Generate and export fake data 🚀 Generate the rows you need for each table and export the data into your database. The beauty of this workflow is that every time you make a software change that requires a change in the database schema, you can re-generate fake data with minimal changes to the code you already wrote. 📚 Learn more about DayZSynthesizer here: bit.ly/41j5ADs 📚 Learn more about the Metadata Creation API Here: bit.ly/3QnPVfX 📚 Learn more about the SDV here: bit.ly/4b858Lu #syntheticdata #fakedata #machinelearning #generativeai