%md # Insurance Claims - Fraud Detection
**Business case:**  
Insurance fraud is a huge problem in the industry. It's difficult to identify fraud claims. IHS is in a unique position to help the Auto Insurance industry with this problem.  

**Problem Statement:**  
Data is stored in different systems and its difficult to build analytics using multiple data sources. Copying data into a single platform is time consuming.  

**Business solution:**  
Use S3 as a data lake to store different sources of data in a single platform. This allows data scientists / analysis to quickly analyze the data and generate reports to predict market trends and/or make financial decisions.  

**Technical Solution:**  
Use Databricks as a single platform to pull various sources of data from API endpoints, or batch dumps into S3 for further processing. ETL the CSV datasets into efficient Parquet formats for performant processing.  

%md
In this example, we will be working with some auto insurance data to demonstrate how we can create a predictive model that predicts if an insurance claim is fraudulent or not. This will be a Binary Classification task, and we will be creating a Decision Tree model.

With the prediction data, we are able to estimate what our total predicted fradulent claim amount is like, and zoom into various features such as a breakdown of predicted fraud count by insured hobbies - our model's best predictor.
      
We will cover the following steps to illustrate how we build a Machine Learning Pipeline:
* Data Import
* Data Exploration
* Data Processing
* Create Decision Tree Model
* Measuring Error Rate
* Model Tuning
* Zooming in on Prediction Data

%md ## Data Import

First download [this csv file](https://raw.githubusercontent.com/jodb/sparkTestData/master/insurance_claims.csv) locally

The data used in this example was from a CSV file that was imported using the Tables UI. [Databases and Tables](https://docs.databricks.com/user-guide/tables.html)

After uploading the data using the UI, we can run SparkSQL queries against the table, or create a DataFrame from the table.  
In this example, we will create a Spark DataFrame.

%md ## Data Exploration

We have several string (categorical) columns in our dataset, along with some ints and doubles.

%md Count number of categories for every categorical column (Count Distinct).

%md 
We have identified that some string columns have many distinct values (900+). We will remove these columns from our dataset in the Data Processing step to improve model accuracy.
* policy number (1000 distinct)
* policy bind date (951 distinct. Possible to narrow down to year/month to test model accuracy)
* insured zip (995 distinct)
* insured location (1000 distinct)
* incident date (60 distinct. Excluding, but possible to narrow down to year/month to test model accuracy)

%md Like most fraud datasets, our label distribution is skewed.

%md We can quickly create one-click plots using Databricks built-in visualizations to understand our data better.

Click 'Plot Options' to try out different chart types.