Do I need to be Dr. Frankenstein to create real-ish synthetic data?

Speaker 1 [00:03]

Hello, everyone. First of all, apologies that I am presenting like this online because of visa hassles that happened at last minute. So my name is Gatha and here's my talk about practical synthetic data sets. And why did I remember the mythical monster of Frankenstein's is because we want our synthetic data sets to be as real as possible. We should be able to fool the models or fool the analytics that we are doing on the synthetic data sets and not have villagers coming to a lab with pitchforks and torches. So a little about myself. I am currently pursuing a doctoral in advanced data mining. And since my specialization is in theoretical privacy, I have been researching in the field of practical synthetic data sets as well. I am an active writer and you can look up my writings on Medium, Senses and Open Mind. So I write about deep learning and explainable AI, private data analytics and MLOps. And I would definitely be happy to connect with you on LinkedIn. So starting with what questions brought you here. So I would make the safe assumption that we are not here to talk about the mythical monster, but about synthetic data sets. So these are the four questions that I would like to focus on in this talk. So what are the types of synthetic data sets and why do we need them? And once we have made a certain synthetic data sets, then what are the metrics that are used to measure that realness? And definitely how you can make synthetic data sets of your own, because definitely you have requirements that are very specific to your field and your problems. So based on how the synthetic data sets are created or synthesized, there are three major types. So the first is the ones that are synthesized from real life data. Second is from assumptions. And third is a hybrid of synthesis from real data and assumptions. So when we talk about synthesis from real data, we use generative models. And very commonly, we use GANs for this. So we fit a model to the real-life data. And then once it's fed, it has sampled the real-life data. Then we generate synthetic datasets from the trained model. So this is a very simple diagram that shows the process. Again, real-world datasets are right-wing problems. They have missing values. The dimensionality could be large. and there could be certain problems with the data acquisition process that introduces errors or discrepancies in the attribute values. So the synthesized data set would also reflect those ambiguities. So what is done in certain cases is that synthetic data sets are created based on assumptions. So what are these assumptions? these assumptions are taken from existing models that have been trained and shared with public or they could be based on assumptions by data analysts so a person or if your team is working on a customer behavior and it needs to analyze and build a model on that so the customer behavior it is a very volatile domain and it changes throughout the year it is a very time-specific behavior for instance sale of ugly sweaters during christmas so your team or the expert analyst in your team they might have a good knowledge about customer behavior specifically for different times of the year or across the years so we can use their knowledge some of the examples are historical trends in financial markets since financial markets are extremely volatile and then statistical distribution of human traffic in an area so again it is a very it is a very dynamic situation and we cannot rely on real life data sets we will have to go about capture the data first and we need to be sure about its integrity so rather than going the more expensive path what we can do is we can model an assumption and then create data sets from those and where exactly does the synthesis from assumptions help when there is very limited real historical data or we have an expert analyst on board or if the situation is noble so the recent years they had shown a very novel situation of the pandemic, COVID-19 pandemic. And initially, as I observed in my country, and maybe some of you also observed it, there was a taboo attached to getting afflicted with COVID. People would do self-medication and try to hide their status. And sometimes if the situation worsened, only then they sought medical help. so in that case some of the data it was missing for the initial time so such an old situation it wanted it asked for data sets but because of the taboo and the limited access there was a shortage on the required data and this greatly hindered research as well so in such cases synthesis from assumptions can be greatly beneficial and we have a mixture of synthesis from the assumptions and real life data so that you can zero in on your domain specific requirements so which one is the best for you so we have these types of synthetic data those generated from non-public real world data sets so the utility can be quite high especially since they are non-public and how is the difference uh brought in by real public data a real public data would be de-identified it would have privacy provisions uh imposed on it so the utility could be greatly hindered by that then we can have an existing model or a process which can use simulations to generate the data sets this again the utility of such data sets would depend on how properly or how truly the existing model had sampled the data and then given the output then there are practical synthetic data sets that are based on analyst knowledge again this is dependent on that person's knowledge and how much they understood the requirement as well as how much they are expert in the domain then there are generic assumptions like we talk about customer behavior over the span of a year so we might assume that ugly sweaters sell a lot during christmas so that's one assumption and i would like to generate some synthetic data on that so utility it could be very low because my assumptions they could be biased they could be too tightly coupled with what my assumptions are so why synthetic data sets there are two main reasons that ushered in the popularity of synthetic data sets first was efficient access to data so as we all know uh data acquisition it is very costly it is dependent on people as well sometimes people would promise that they would answer your survey but they don't we all know that so definitely access to data And secondly, there are secondary purposes to data sets. For instance, if you had collected health data for cancer patients and you had used it to drive your medical research, now there arose a secondary purpose of checking hospital-related logistics. so how do we need to decide on certain facilities that need to be provided to cancer patients so that is the secondary purpose and when we talk about such purposes there are very stringent regulations like HIPAA so HIPAA it has put on very strict restrictions because any private health information if it is identified even by a single field for instance your phone number or email then it would prohibit sharing of that data set. So this is a major hurdle for sharing of the data sets and therefore the answer was found in synthetic data sets. They were not seen to be prone to such disclosures and pretty much in sync with GDPR regulations. Important benefit of synthetic data sets is that they enable better analytics in certain cases where data collection could be costly or ethically restricted. For instance, we want to drive analytics on data specific to children or certain ethnicities who are not technically advanced, so they do not understand the implication of sharing their data. While there are regulations that specify how the individuals need to be aware of what would be the certain outcomes if they share the data. So in such cases, it is better to go with the safe option of synthesizing data. Then annotation of real-world data can also prove to be a costly process, especially if you need to employ a team or a certain tool to achieve that. Then synthetic data has also proven to be of great use when we do proxy analysis to check the feasibility of a certain model or analytics. So to save on the cost of data acquisition first certain test and proxy test can be done through synthesized data and if the models are found to be performing well then we can go ahead and allocate budget for data acquisition. So there are some specific domains that greatly benefit from synthetic data when we come to manufacturing and distribution now we have robots that are being powered by aiml they do this location and picking of objects that are in different positions and stacked up differently so a very popular example is the amazon picking challenge where they pit teams against each other to device complex models where the models have been trained to detect different permutations of positions they should be able to handle different lighting scenarios so rather than using people to carry out all these permutations it makes sense to synthesize such images so this is one example image it's a synthetic image of dominoes placement that was simulated by nvidia graphics rendering engine So rather than having someone place dominoes in all these inexhaustive combinations of domino spaces, we have this engine that is quickly generating images for you. Healthcare definitely has gained a lot from synthetic data because individual information is very critical and demands legality. so all these uh requirements uh they cause a lot of cost intensive operations like privacy provisions and the identification so uh based on the sensitivity of afflictions and on the information that we need to have individuals of individuals we should go for synthesis in those cases additionally uh some medical data sets can be complex like images so for instance we have retinal images data set and now the team wants to do an analysis for diabetic retinopathy so in this case rather than going around and approaching diabetic patients and then checking if they are willing to share the images of the retinas going through all those tests it is better if we synthesize such images from existing retinal images add some clouding and the problems that typical of the affliction a synthetic cancer registry being maintained by public health england is one very commonly available cancer individual data that you can check out in the link then in some cases we need to do a rapid evaluation of a technology for instance keybridge semantics a boston based firm they created a graph database for healthcare cases and they wanted to check how the this pilot project would work before starting a actual data acquisition so they use synthesized data to check the performance of this pilot and once the performance was found to be promising they went ahead with data collection when we come to financial services again there is a major need for privacy provisions and the historical market data is vast and it is it could be expensive based on the access restrictions that have been placed by the custodians and the owners so uh and there are benchmarking uh requirements as well so we have a risk modeling software and hardware that need to be evaluated for the real-time performance so stack a to benchmark you can check out the link this is being used to benchmark the performance of such modeling software. When we come to transportation a very interesting use case is self-driving cars that are being driven by AI ML. So these models they are trained and tested on images that were captured from LiDAR sensors and cameras mounted on top of cars and those cars they were driven around a certain area. So you see the problem that the geography is restrictive it is very typical of this that place and the driving and walking behavior of people of that area so again we are going by the assumption that all roads and all driving conditions would be similar to what had been recorded which is definitely not the case so what we can do is synthesize further images for different topography and different driving behaviors to test out these autonomous driving solutions and hackathons and datathons that I think many of you must have taken part in it. So again, to circumvent legality and privacy provisions and the whole cost of acquisition, these hackathons, they sometimes rely on synthetic data sets to pose a problem and pose these events. so how do we measure the realness of these frankenstein data sets you created a frankenstein and you wanted to mix in with the real world without having villagers knocking on your door and coming with their pitchforks and torches so what do we do we use some metrics to check how nicely a synthesized data is following its seed there are four ways of comparison the statistics let's start with the univariate statistics so we plot the trends followed by one feature between the real and the synthetic data sets if we go by the looks of it the left one is a very good case now the synthetic data and real data pretty much in sync while the right one is at a chaos you want to ask the model what they were thinking But pictures are again not everything. We need to put numbers to these metrics. So what we do is we compute Hellinger distance, which is the measure of probability. And it indicates whether there is a difference between the two distributions of one variable. So it's a probabilistic measure. So that means that the differences have been scaled between the values 0 and 1. coming to bivariate statistics comparison we take pairs of variables between the two data sets under comparison and it is definitely driven by the type of variables that we have under consideration so if we have a continuous variable pair we go for Pearson coefficient if the pair involves continuous and nominal variables then we go for multiple correlation coefficient and so on once you have computed the correlations you can again plot a heat map of the computed values here we see the problem typical of synthesizing from real world data sets so while lighter shades the ones tending to white they show great promise that the synthesized data was close to the real one but all those gray boxes they show that correlation could not be computed because of missing values or low variability offered by the feature. So when comparing multivariate statistics, we are playing the game of you are it. So what we are doing is we take each feature, we say you are it. And what does that it means? It is the target variable that the classification model would try to find or output. So it is a very exhaustive process because we are going through each feature and making it the target variable for that run we can definitely boost the performance through gbms but again it is quite exhaustive and computation heavy next we come to distinguishability so what distinguishability are we aiming at the distinguishability between a particular observation that belongs to a real record or the synthesized record so what we do is we build a binary classification model and what is the target variable target variable is a new field which could be 0 or 1 that we assign to each observation of the data sets so if it's a observation of a real record it will have one extra variable assigned with a value of 1 and if it's a synthesized record the observation would have this new value zero and these uh these records with this extra variable they would be used to train and test the binary classification model the intent is to check whether the classification model can find uh or it can predict properly if a particular observation belong to a real record or a synthetic record so for this metric we use the value of propensity score which is the probability of each of the prediction so in this case we had assigned one for a real record so propensity score which tends to one means that the observation belongs to the real record and we can definitely do the assignment and vice versa also so then you have to be careful how you watch this propensity square propensity score so great we know the basics of synthetic data sets you want to get started you want to make synthetic data sets of your own based on what problem you have at hand so pi so pi db gem is a very popular library that you can use to create er databases yeah it is data frame friendly so what database you create can be easily read through panda's data frame and you can do the further processing under the hood it's using Faker, which is also a very popular library. You can fill in the fake data in your data sets if they're missing values. And you can definitely use Faker on its own as well to create a data set. And then you might have scenario-specific requirements. For instance, you want to simulate a music streaming event which had different genres and different kinds of individuals who were part of that event where different age groups and genders and their preference for specific genres so such short-term scenarios they can be synthesized using trumania it is a very exciting library in fact you can also simulate customer behavior for a specific period of time so just to see that how if you want to simulate how people buy certain item during the first months of the year or during the middle of the year so trumania is definitely the library to use then we have data synthesizer which provides a web-based user interface where you can synthesize a real data set additionally you can add differential privacy to the seed data set the real one and then go on with the synthesis so you will be working on a private real data set and synthesizing from it which is a very good thing to do especially if you're working on critical data sets then you can create regression and classification specific data sets as well using senpai so when we talk about synthetic data vault it is a development environment rather than a library as such you will get the environment conducive to creation of gans and distribution functions that you can further use for data synthesis uh definitely time series is one kind of data that you would want to synthesize at some point so nike's time series generator is a very popular way to go about it so great uh let's capture some lightning and create a frankenstein so i have this piece of code where we are using numpy and scikit-learn to create some synthetic data. The link to this notebook has been shared in the PPT as well if we would be sharing it further after the presentation. So definitely the setup requires import of certain libraries. This is the seed value 11. You can go about it and play with the seed value. So first we would synthesize univariate distributions of different types, like uniform, normal, exponential, lognormal, chi-square, and beta. So these are the different distributions, or this is fake data. Coming to specific requirement of classification, but I want a synthetic data set for a classifier. So for this, scikit-learn has these functions. it has make classification make classification function has this parameter class step which specifies what would be the spread of the classes the target variables so we have two features class step is of different values 0.1, 1 and 10 will be generating 1000 samples and these are the three plots for different spreads so for point one you can see the two classes are pretty much overlapping with class f1 there is a little difference and with class f10 they are very clearly demarcated now we'll come to a multi-label classification where we want more than two classes so again now this function is make multi-label classification where again you can specify the number of features the number of observations and labels where labels have kept as number only number of classes here is four so for different labels we have these plots so similarly scikit-learn can be used to generate data sets for regression as well as clustering problems so this is all for today's talk I have curated the sources as well for you that I have kept at the end of the slide and if the BBTs are shared then these resources can be used by you for the knowledge so thank you for your time and I hope that you could take something new from here and looking forward to your feedback and questions Thank you.

Speaker 2 [25:51]

All right. Gata, can you hear us? Can you hear me? Yeah? Awesome. Yes. All right, cool. I'm going to read a couple of questions that came in on Slido. The topmost is, have you considered cuplas for modeling testing for equality of multivariate dependencies?

Speaker 1 [26:12]

I didn't get a question. Have you considered?

Speaker 2 [26:13]

copulas sorry if I pronounce this incorrect have you considered copulas for modeling testing of equality of multivariate dependencies no I have not ok

Speaker 1 [26:31]

Thank you for this one.

Speaker 2 [26:33]

Next question is, how do you preserve the connection between the label and the features as well as the joint distribution of the features in a synthetic data set?

Speaker 1 [26:46]

Okay, so this is pretty much a characteristic of how we are synthesizing from the seed data. So the libraries that we are using, they are preserving these relationships between the features and the distributions.

Speaker 2 [27:04]

All right, thank you very much and the last question is how would you approach showing a realness in synthetically generated image data

Speaker 1 [27:15]

So, in images, there was this research that I read in which we also modeled another shadow adversary kind of model that was also checking the classification of the images. So, we have the main synthesized images, the main classification model, and one shadow adversary that was checking how well it could identify the fake and the real images. Thanks for watching!

Speaker 2 [27:46]

Okay. And another question. Besides being distinguishable, do you have recommendations for other metrics that measure the level of privacy gained over the real data?

Speaker 1 [28:00]

Okay, the privacy provisions. Synthetic data sets, they are definitely prone to disclosures as well. So the privacy measures, if you have to mark them against the utility, entropy can definitely be computed for that. We have to be very mindful that utility is not disturbed if we are enforcing privacy as well. So my suggestion would be starting with privatized data sets, the seeds that we are using and then carry out synthesis on them.

Speaker 2 [28:33]

Awesome, and in your experience does a model trained with synthetic data instead of real data lose accuracy?

Speaker 1 [28:43]

I used one, sorry.

Speaker 2 [28:44]

If it loses accuracy

Speaker 1 [28:48]

So the intent is that we need to fool the model. So I have run some pieces of code. I have trained a model that used a combination of real data and fake data. And the metric used was RMSE. And I achieved pretty good results in terms of very low RMSEs with that combination as well.

Speaker 2 [29:13]

All right, okay, that would be all questions if there's no more questions for the room Then that's a wrap. Thank you all very much