Jorion - Backtesting Questions

[mathman]

Q1. How are they mathematically getting the values for k under the Basel penalty zone to go from 3 to 4 for a 250 day 99% CI when the number of exceptions goes from 5 to 10. Or is it something that has been set by Basel?

Q2. There is an example in Jorion where they have found that when p = 0.01 and T = 250 the number of exceptions above 4 is 10.8% (when modeled correctly - what does that mean - to model correctly? and how did they calculate 10.8%??)

Then they have shown that the number of exceptions when the model is calibrated incorrectly, that is, when p = 0.03 instead of 0.01, the value calculated is coming to 12,8% - meaning that we will not reject the incorrect model more than 12.8% of the time. (so how did they get 12.8%? and what does it mean to incorrectly model? what are they meaning to say when they talk about 0.03 vs 0.01 in plain english?)

Thanks.

 

[David Harper CFA FRM]

Hi mathman

1. k=3 the floor and k = 4 (the upward adjustment) were set by Basel (back in the 1996 amendment). I am not aware of a strict mathematical basis, although I've read somewhere some commentary that showed how *3 implicitly moved the confidence up from 99.0% to something higher.
2. The VaR backtest applies a binomial distribution (since exceptions [loss in excess of VaR] either occur or do not). The first instance (Jorion Figure 6-2) illustrates a 99% VaR model that is accurate (good), such that binomial p = 1%. Under binomial with T= 250, p = 1%, P[X > 4] = 1 - P(X <= 4) = 1 - binomdist(x = 4, trials = 250, p = 1%, true = CDF) = 10.8%; i.e., the probability of observing 5 or more exceptions when the model is indeed correctly calibrated with mean of T*p.
   
   The second instance is different because it illustrates a "bad" or "incorrect" model, so it uses p =3% when we think p =1%; i.e., our model is 99% VaR but it actually is miscalibrated so it performs with p = 5%. So, Jorion [like Basel] could also use p = 4% or p = 5% etc etc. But to illustrate a bad 99% Var with actual p = 3%, 12.8% is the CDF proba [X <=4 | T=250, p=3%] = 12.8%; ie, the probability of observing 4 or fewer exceptions when the mean is actually 250*3% and mistakenly accepting the model as good (Type 2 error b/c the null is "model is good").

 

[mathman]

Hi David,

Thanks for the quick replies.

Soln 1. tells me that I neednt be bothered about how to calc. k values as these are set values - thanks for the info.

Soln 2. took me back to high school math. I did the math and got the numbers! ( Umm.. isn't the number of trials supposed to be 250, not 10? I got 10.91% by the way).
But I do want to ask ( correct me if I am wrong -this is my understanding): Basel allows the banks to have 1 to 4 exceptions. The banks take (say) a p of 5% incorrectly. So they feel that they are cool with making 5 to 20 exceptions now. But Basel allows them only 4 max (for green). What happens next?

Thanks again.

 

[David Harper CFA FRM]

Hi mathman - (yes, 250 trials, not 10, i fixed above). Although Basel does not specify the particular VaR approach (a bank has latitude, most employ HS), market risk must be calibrated to 10-day 99.0% confidence (a "quantitative standard"). To employ its own internal model (IMA), the bank must use a 10-day 99.0% VaR model, it cannot charge market risk capital against (eg.) 95% confidence. Further the bank must conduct the backtest. 10/250 or more exceptions enters the "red zone" and automatically imposes +1 to k = 4, as a minimum consequence. The model is presumed bad and Basel wants the supervisors to investigate, with the presumption that the internal model is not qualifying (e.g., supervisor could revert to standardized approach). This is first pillar basel (the rules); much of the practical implication (what happens next?) refers to the second pillar: basel delegates much of the actual inspection/reaction to national supervisors. This is sometimes neglected in study: first pillar rules are just minimum standards. Basel is a framework across nations and bank types/sizes (and many would argue it fails even that), so second pillar is the "load bearing pillar" where realistically supervisors must have some autonomy. Thanks,

 

[mathman]

Thanks for the info. It makes the process clear now.