Collaborative ML research projects within a single cloud environment

As one of the largest banks in Indonesia and Southeast Asia, Bank Rakyat Indonesia (BRI) focuses on small-to-medium businesses and microfinance. At BRI, we established a Digital Banking Development and Operation Division to implement digital banking and digitalization. Within this division, a department we call Digital BRIBRAIN develops a range of AI solutions that span customer engagement, credit underwriting, anti-fraud and risk analytics, and smart services and operations for our business and operational teams.

Within Digital BRIBRAIN, our AI research team works on projects like the BRIBRAIN Academy — a collaborative initiative with higher education institutions that aims to nurture AI and ML in banking and finance, expand BRI’s AI capabilities, and contribute to the academic community. The program enables students from partner universities to study the application of AI in the financial sector, selecting from topics such as unfair bias and fairness, explainable AI, Graph ML, federated learning, unified product recommendations, natural language processing and computer vision.

Based on our long history and work with Google Cloud, with some Vertex AI technology implemented in other use cases, we selected its products and services to provide a sandbox environment for this research effort with partner universities. This research covers a range of use cases and concepts, including the following:

1. Fairness analysis on credit scoring research in banking

Industry-wide, banks and other financial institutions use credit scoring to determine an individual’s or organization’s credit risk when applying for a loan. Historically, this is a manual and paper-driven process that uses statistical techniques and historical data. There is considerable potential benefit to apply automation to the credit scoring process, but only if it can be done responsibly. 

The use of AI in credit scoring is a noted and well-documented area of concern for algorithmic unfairness. Providers should know which variables are used in credit scoring AI models and take steps to reduce the risk of disparate model outputs across marginalized groups. To help bring the industry closer to a solution where unfair bias is appropriately mitigated, we decided to work on fairness analysis in credit scoring as one of our BRIBRAIN Academy research projects.

Fairness has different meanings in different situations. To help minimize poor outcomes for lenders and applicants, we measured bias in our models with two fairness constraints, demographic parity difference and equalized odds difference, and reduced unfair bias with post-processing and reduction algorithms. As a result, we found that the fairness of demographic parity improved from 0.127 to 0.0004, and equalized odds from 0.09 to 0.01. All of the work we have done thus far is still in the research and exploration stage, as we continue to discover the limitations that need to be navigated to improve fairness. 

2. Interpreting ML model decisions for credit scoring using explainable AI

Historical data is used to train a model to evaluate the creditworthiness of an application. However, the lack of transparency in these data can make it challenging to understand, and the ability to help others interpret results and predictions from AI models is becoming more important.

An explanation that truly represents a model’s behavior and earns the trust of concerned stakeholders is critical. With explainable AI, we can get a deeper level of understanding of how a credit score is created. We can also use the features we built in the model as filters for different credit scoring decisions. To conduct this research collaboration, we needed to leverage a secure platform with strict access controls for data storage and maintenance.

3. Sentiment analysis of financial chatbots using graph ML

Chatbots are computer programs that simulate human conversations, with users communicating via a chat interface. Some chatbots can interpret and process users' words or phrases and provide instant preset answers without sentiment knowledge. 

Unfortunately, responses are sometimes taken out of context because they do not recognize the relationship between words. This means we had to represent chatbot data that can learn relationships between words through preprocessing using graph representation learning. These methods help to account for linguistic, semantic, and grammatical features that other natural language processing techniques like bag-of-words (BOW) models and Term Frequency-Inverse Document Frequency (TF-IDF) representation cannot catch. 

We built a sentiment analysis model for financial chatbot responses using graph ML, allowing us to identify which conversations are positive, neutral, or negative. This helps the chatbot avoid mistakes in categorizing user responses.    

Deploying data warehousing, ML, and access management tools

Google Cloud met our needs for these projects with infrastructure and services, such as its cloud data warehouse BigQuery and  its unified machine learning (ML) development platform Vertex AI, which offers a range of fully-managed  tools that enabled us to undertake our ML builds.

We also used Vertex AI Workbench, a Jupyter notebook-based development environment, to create and manage virtual machine instances adjusted to researchers’ needs. This enabled us to perform data preparation, model training, and evaluation of our use case model. 

Using the structured data stored in BigQuery, we were able to write our own training code and train custom models through our preferred ML framework. Furthermore, we employed Identity and Access Management (IAM) to deliver fine-grained control and management of access to resources.

The general architecture we used to support each research topic is below:

We loaded masked research into BigQuery and gave researchers access to Vertex AI for specific BRIBRAIN Academy projects, assigning a virtual machine on which to conduct research. They could then use Vertex AI Workbench to perform the pipeline steps illustrated above in Vertex AI Workbench and access required data in BRIBRAIN Academy projects via BigQuery. 

To build and run our ML solution efficiently and cost-effectively, we limited the resources available to each user. However, Vertex AI enabled us to modify instance resources to accommodate cases where significant data volumes were needed to create a model. 

At the same time, Google Cloud data security services allowed us to protect data at rest and in transit from unauthorized access while creating and managing specific access to project data and resources. We provided specific access to researchers through BigQuery and notebook custom roles, while developers received administration roles.

Undertaking research projects within a single platform

With Google Cloud, Digital BRIBRAIN now has the power to explore use cases from BRIBRAIN Academy and apply lessons learned in live business projects.

For example, we have already used research around AI explainability to help us develop end-to-end ML solutions for recommender systems in our branchless banking services, known as BRILink agents. We also built a mobile application containing recommendations with AI explanations. In an environment where many users are unfamiliar with ML and its complexities, AI explainability can help make ML solutions more transparent so they can understand the rationales behind recommendations and decisions.  

With our success to date, we plan to evolve our ML and data management capabilities. At present, we use BigQuery to store mostly tabular data for training and building models. Now, we are expanding these capabilities to store, process, and manage unstructured data, such as text, files and images, with Cloud Storage. In addition, we plan to monitor app usage using reports generated through Google Analytics for Firebase with some of the ML solutions available in our web-based applications. 

Google Cloud gives us the ability to store our data, build and train ML model workflows, monitor access control, and maintain data security — all within a single platform. With the promising results we’ve seen, we hope to be able to tap into more of Vertex AI capabilities to support ongoing developments at BRIBRAIN Academy.