Risk Management, PRMIA

October 07, 2010

Big Data and Risk Management

Big Data is now all the rage. Even the Economist has a writeup on the subject. Data, once exclusively the province of quants and geeks, is enjoying visibility in the executive suite like never before. Data has always played an important in risk management. The importance of good data inputs to quantitative techniques has never been in question. This need has been further exacerbated by new risk management and regulatory requirements (Dodd-Frank Act, Basel III etc.).

Exciting change is also occurring on the technology front. In the past 2-3 years a whole new class of open-source big-data manipulation technologies have come of age. These have been driven largely by the needs of dot-com companies to process vast amounts of information being generated from web-site activity, but they are finding new and interesting applications beyond their traditional uses. This class of technologies has been dubbed "Big Data" by the popular press. Driven by the evangelistic fervor of its proponents, the press has taken to making mythical claims about this technology including the imminent demise of traditional database technology.

At the same time, there is a huge amount of interest in precisely that so-called legacy technology, driven by intense M&A activity in the sector (note the hyperlinked article is prior to IBM's recent acquisition of Netezza) and consequent high-flying stock prices of vendors in that space. Clearly these are contradictory statements. The billion-dollar question is: who's right - the Mr. Market or the geeks?

This is a great topic to think about. I will examine all these developments in some detail in the next few blog posts, and especially focus on how they apply to risk management applications.

Risk Management and Big Data

Let's go back to basics and consider why risk management cares about data. At it's core, risk management tries to analyze the historical record to make predictions about the future. These predictions are then woven into a risk management process to mitigate their results. This sort of analysis requires access to detailed historical data. It is not sufficient to analyze historical results, but also have access to details of historical transactions and outcomes so that fresh new analytical techniques may be applied, even those that did not exist at the time when the transactions were recorded.

This need for data has increased manifold since the credit crisis of 2007-08. Now risk management is a matter of serious focus in boardrooms, and pithy assumptions about risk ("If it's rated AAA it must be ok") are no longer accepted. Risk managers are having to perform deep analysis as their conclusions are being challenged like never before. Challenges are coming not only inside financial institutions, but also from regulators. The recently enacted Dodd-Frank Act in the US is only one example of a world-wide trend of focus on risk management.

What sort of new requirements are risk managers having to deal with? A sample is shown below:

• Complete view of risk: It is no longer acceptable to allow a siloed organizational view to prevent the top management and regulators from gaining a comprehensive view of the risks being borne by the institution.
• Risk-integration: Erstwhile techniques used disaggregated views of risk types (credit, market, liquidity etc.). The recent crisis has proven that risk types are fungible - credit risk of an underlying reference entity can quickly transform into the liquidity risk of diverse classes of bonds and derivatives that cannot be funded or sold.
  
• Stress testing: One of the less-known outcomes of the SCAP Program completed early 2009 was the realization that banks were sorely unprepared for repeated runs of even one stress-test, much less address the expected requirements of administering several stress-test scenarios every quarter.
• Reacting to Changing Situations: In the height of the crisis, hitherto unanticipated types of risk began to take center stage. Exhibit A on this list was settlement risk, which is not often addressed in a systematic manner, but became hugely important following the accidental transfer by German bank KfW of 300 million euros to the bankrupt Lehman Brothers. All of a sudden it was not only important to understand outstanding exposures, but also payment and settlement obligations.
  

The list can of course go much longer. It is becoming clear that risk analytics is not only becoming more important, it is also getting more diverse and tailored to the needs of individual financial institutions.

Before we dive into the various technologies to address these challenges, let's look at some common attributes of data that risk management requires:

• Detail: It is no longer sufficient to analyze aggregate data. Detailed transactional and position data is a key component of risk analysis so that ad-hoc aggregates appropriate to any market situation can be created.
  
• Complete: Risk officers now know that notional size of a position is no guide to the amount of risk contained in it. The goal now is to achieve 100% coverage of all positions of the firm, no less.
  
• Timely: This has always been a requirement, but has now been exacerbated by rapidly moving markets. Bluntly put, firms are now willing to pay big bucks for up to-date analysis.
  
• Integrity: The correctness of data is of course table stakes in risk management - garbage-in-garbage-out.
  

While these attributes characterize input data, data processing is also critical. Analysis of risk data can be classified in the following ways:

1. Data Processing: Data can be transformed via calculation to create derived values. VaR calculations and Monte Carlo simulation are a good example - risk analysis contains a plethora of such types of processing.
   
2. Ad-hoc Querying Data: Risk managers need to interrogate the base and derived data from various perspectives in order to form and prove (or disprove) theories about emerging threats.
   
3. Canned Reports: Periodic fixed reports are of course a staple of risk management and are depended upon by management and regulators alike.
   

All these types of analyses are confusingly lumped into the category of risk analytics - making it easy to fall into the trap of viewing a particular technology as the cure-all only to realize that it only addressed part of the problem. In an attempt to clear up this confusion, my next few postings will examine how emerging data management technologies can be best used for risk analysis.

Stay tuned.

Posted by dkrishna at 03:02 AM | Comments (2)