How capital markets can prepare for the future with AI

Editor’s note: This post originally appeared on Forbes BrandVoice.

In capital markets, the stakes have been raised for participants to establish value, win loyalty, and expand their share of wallet. An organization’s data analytics capabilities, combined with artificial intelligence and machine learning, can open new opportunities in these areas. But many organizations are still using data strategies from the past, which limits their ability to harness data to its full potential and make the right business decisions. Without the ability to accurately predict business outcomes with the help of AI, market makers are left to rely on hunches and educated decision-making when predicting the unknown.

Firms are increasingly recognizing the benefits of technology, and partnering with modern tech providers is key to realizing those benefits. But challenges still exist for firms looking to deploy ML at scale. Below, we’ll look at some of those challenges, along with tools and best practices that can help capital markets firms adopt and benefit from AI and ML strategies. 

Challenges in the data-to-ML journey

At a high level, the challenges faced in capital markets when performing AI are similar to other industries. The first set of challenges comes with the data itself. Unstructured data accounts for 90% of enterprise data, and many enterprises face the limitations of on-premises and legacy applications that don’t work well with newer cloud-based tools. Also, a high number of data silos spread across capital markets are common due to growth through acquisitions—a time-consuming distraction that limits efficiency and decision-making. Data science is not hamstrung by the velocity of messages, nor the volume, but by the huge variety of disparate data sources.

Other challenges include the views and varying levels of resistance regarding the value of data by various stakeholders within the enterprise; the restrictions of regulatory environments; and the limited cloud skills of an enterprise’s IT teams. ML operations can also be challenging as firms enter this emerging technology area.

Adopting and benefiting from AI and ML strategies: Tools and best practices

1. Before you perfect AI, get good at analytics
Effective AI and ML depend upon a strong and flexible data analytics platform, which first may need some rearchitecting of its infrastructure. Without a strong core data infrastructure, it’s hard to perform data science in production. With enterprises that have adopted traditional data analytics platforms that live on local servers, challenges abound—and the blue dollar costs (those charged back within the company) go far beyond software licensing. These enterprises have to expend costs and resources on monitoring, performance tuning, upgrading, resource provisioning, and scalability. Business-critical data sources may not be easily accessible by data scientists, blocking business-critical decision-making. All of these obstacles leave less time and room for gleaning analysis and insights from the data.

With a serverless, cloud-based data analytics model, the vast majority of infrastructure maintenance and patching is handled by the cloud provider. This enables your data team to devote more time and resources to analysis and insights. Highly performant and integrated cloud technologies can help enterprises overcome data silos, establish a single code base, and contribute to a more collaborative workplace culture. They can also be designed to provide more real-time insights—an invaluable building block of ML and AI. In short, effective core data infrastructure is a competitive advantage over other organizations that remain stuck in silos and servers. 

2. Get started by prioritizing a business goal
In the past several years alone, a number of common use cases for AI have arisen in the capital markets sector. Here are some specific examples, and how AI can help:

• Dynamically learn how best to place orders across venues with algorithmic execution.

• Recognize potential triggers for unscheduled events with predictive data analytics to forecast events.

• Generate multi-dimensional risk and exposure data analytics with real-time risk analysis.

• Use ML to help gain insight into the selection process via algorithms for asset selection. 

• Determine client needs/opportunities using social media sentiment analysis. 

• Build systems that can respond to client inquiries via speech-to-text natural language processing. 

• Extract key data from unstructured or semistructured documents with natural language document analysis services. 

• Generate performance and financial data commentary reporting with natural language generation for document writing. 

• Identify complex trading patterns in large datasets with market abuse and financial crime surveillance. 

Though it’s tempting to focus exclusively on the benefits that tech can bring to data analytics, the immediate opportunity for how enterprises can fully benefit from AI rests in how humans and AI can work together. ML-based data analytics is more powerful when paired with human judgment and intuition. Recent advancements in tech have made computers faster, data storage cheaper, and access to algorithms more democratized.

But human experience and judgment can contribute to and expand upon accurate, insightful data analysis, whether that be in medicine or in financial markets. Model explainability and fairness are concrete examples of where human experience is critical to successful AI (more on that below). When designing an AI system for use cases like the ones listed above, don’t divorce it from the benefits of human wisdom. 

3. Structure your team for better data decisions
Finding, retrieving, and preprocessing data can be the most time-consuming part of building ML models. Over 80% of model building effort goes here. This challenge is not unique to financial services, but addressing this challenge is a necessary prerequisite for ML, and affords a competitive advantage. Structuring your organization and internal teams to tackle this challenge will increase your odds for success, but will require some planning and careful thought. 

Simply put, the purpose of a data science team is to facilitate better decision-making using data. Keep this in mind when deciding how to best structure your data science and AI/ML teams, as well as who they’ll be reporting to. It's also important to consider where your organization currently sits in its data and AI journeys. Consider culture, size, and the ways the company has grown. Is your enterprise centralized or decentralized? Is it federated? Do you employ consultants?

When defining team roles, consider how your flow of data is structured, and where those roles would be of most efficient use. Also, don’t limit yourself—different roles don't necessarily require different employees. People can perform different roles, as long as the roles are clearly defined. 

4. Understand the concepts of explainability and fairness
There are two important considerations to keep in mind when structuring your organization for data analysis and AI. The first is explainability. We want AI systems to produce results as expected, with transparent explanations and reasons for the decisions they make. This is known as explainability, a high priority here at Google, as well as a growing area of concern for enterprises when it comes to designing their AI systems. Explainability increases trust in the decisions of AI systems, and a number of best practices have evolved to ensure that trust. These include closely auditing your work and data science processes; monitoring what’s called “model drift” (also referred to as “concept drift”); including accuracy metrics; and ensuring reproducibility of features. 

Fairness is another important topic in AI. An algorithm is said to show fairness if its results are independent of certain variables, especially those that may be considered sensitive. These include individual traits that shouldn’t correlate with the outcome, like ethnicity, gender, sexual orientation, or disability. An accurate model may learn or even amplify problematic pre-existing biases in the data based on those traits. Identifying appropriate fairness criteria for a system requires accounting for UX, cultural, social, historical, political, legal, and ethical considerations, several of which may have tradeoffs. 

Best practices for fairness include: 

• Designing your model using concrete goals.

• Monitoring goals through time for your system to work fairly across anticipated use cases—in a number of different languages, for example, or in a range of different age groups.

• Using representative datasets to train and test your model.

• Using a diverse set of testers.

• Thinking about the model’s performance across different sub groups.

Building your roadmap for the future with AI/ML 

Capital markets’ rich history of using cutting-edge technology now includes AI to open new opportunities in the sector. Foresight and planning will ensure the best results from ML and AI—they shouldn’t be an afterthought for your organization. That means building a strong core infrastructure for data analysis first, planning the structure of internal teams that will use data and AI, and using flexible, cloud-based tools to optimize results.

When introducing new AI/ML strategies, IT leaders must ensure that they integrate and fit with existing modernization efforts, as opposed to being a bolt-on afterthought. This will lead to a true integration of AI/ML and business.