Choosing the Right Infrastructure
Unlike areas like computer vision or natural language processing, which require expensive GPU infrastructure, training risk models doesn’t demand as much computational power. A single Amazon Web Services (AWS) EC2 instance with strong CPU capabilities and sufficient memory is often enough. This setup typically costs between $10 and $20 per hour and is only needed during the training phase.
Using multiple machines in a cluster can complicate things. For risk models, it’s more effective to use just one instance to ensure accurate results. Our tests with LightGBM training within Spark clusters showed worse results.
Data Requirements for Training
When training a risk model, having the right amount of data is crucial. Too little data won’t let the model learn effectively, while too much can lead to complications and require larger infrastructure, which doesn’t always lead to better results.
In the consumer lending space, data can be gathered from credit bureaus, which hold information on around 300 million individuals in the U.S. This data is updated monthly to reflect new credit lines and payment histories. AWS S3 is a cost-effective solution to store and access this vast amount of information. Data scientists often store large datasets in Parquet file format (e.g., 10GB files), allowing quicker access and processing.
Direct access to databases or data warehouses like Snowflake is too slow for training models. Instead, data is exported to AWS S3, where the model can quickly access it. Using Python libraries offers more flexibility than relying on SQL queries, especially with datasets of over 30 million records.
The Training Process
How long does it take to train a model with this setup? Processing around 3 million records with thousands of features typically takes 40 to 60 minutes. Training a model involves several steps:
• Model Analysis: Evaluating performance.
• Model Checks: Ensuring everything works as expected.
• Cross-Validation: Testing the model’s reliability.
You might start with thousands of features, but LightGBM helps narrow it down to a few hundred by identifying the most effective ones. This process requires data scientists to run numerous experiments and fine-tune hyperparameters (the settings that control how the model learns).
Ensuring Reliability
How can we ensure the model delivers reliable results? Cross-validation is key. The dataset is split into five parts—four for training and one for testing. This process is repeated five times to validate the model on each portion of data. It’s also crucial to check for overfitting, which occurs when a model performs well on training data but poorly on new data.
Compliance is vital. Lenders must ensure their models adhere to regulations, such as non-discrimination laws. The model must be able to explain why it rejected a specific application, showing which features influenced the decision. Building a model can take up to six months because it involves multiple iterations to fine-tune and ensure accuracy and reliability.
The Build and Deployment Process
Building the infrastructure and deployment pipeline for a risk model can also be time-consuming, often taking around two months for the initial setup. However, once this groundwork is laid, the hard part is over.
When the model is ready, it can be deployed to AWS SageMaker or a similar platform, allowing easy interaction via a simple API (a way for software applications to communicate). Once deployed, the model can process live requests in milliseconds.
Conclusion
Creating a robust infrastructure for risk model development requires careful planning and execution. The journey from data collection to deployment is complex, but with the right approach, it can lead to significant improvements in the lending process and better risk assessment. Today, developing a risk model is not overly complicated. Beyond a skilled team of data scientists and analysts, the key to success lies in obtaining high-quality training data.
If you’re looking to enhance your risk models or streamline your lending platform’s decision-making, our team can help you build, deploy, and refine cutting-edge machine learning solutions. Get in touch with us today to explore how we can support your goals and ensure regulatory compliance in your lending operations.
