Confidence Interval - Monte carlo

[Laura517]

Hi,

I am having difficulty in solving the following type of questions and cannot understand the solution posted. Could you please explain the solution for the following questions?

400.2. Peter the Analyst has generated 800 independent scenarios of future single-period portfolio values. He observes the mean (average) of his simulated output distribution and determines a 95.0% confidence interval with a length of approximately $300.00; i.e., length is the difference between the upper and lower bound of the confidence interval. Peter's manager wants him to increase the accuracy of his estimate of the population's mean by reducing the length of the confidence interval to about $60.00. How many scenarios should Peter run?

a. 800; no change in trials but increase the confidence level
b. 4,000
c. 7,200
d. 20,000

P1. T2. Panchamanova.

4.
A Monte Carlo simulation consisting of 100 replications returns a 95% VaR quantile of 1.645 (as expected) with a standard error of 0.40, such that the confidence interval for the VaR quantile is [1.245,2.045]. This standard error is deemed too high. How many replications are approximately required in order to reduce the confidence interval to [1.545, 1.745]?

a) 200
b) 400
c) 800
d) 1,600

 

[David Harper CFA FRM]

Hi @Laura517

Right, these are part of a group (theme) that share a single idea which is (if we delegate the deep-dive for later): in each of our relevant cases, the standard error scales by 1/SQRT(n), where (n) is the multiplier on the sample size (or number of trials, in MCS)

There is a standard error for each statistic (sample mean, sample variance, sample skew/kurtosis, VaR, any quantile estimate) but it's a standard deviation; and in our relevant statistics (400.2 is SE of sample standard deviation; your next example is SE of a quantile), the variance scales with (1/n) so its SE scales with the sqrt(1/n). In the case of a sample mean, the SE is SQRT(sample variance/n), such that quadruple the sample (i.e., *4) will halve the SE by multiplying by SQRT(1/4) = 1/2. The statistics vary but in each case (n) is in the denominator such that (1/n) is how the variance of the estimate scales, such that the standard error scales with SQRT(1/n), even for VaR!

In this way, we can say that (eg) double the accuracy is achieved by multiplying the SE by 1/2, which requires a sample multiplier of 2^2 = 4 so that sqrt(1/4) = 1/2; to triple the accuracy, we need SE to multiply by 1/3 and thusly our sample to multiply by 3^2 = 9 so that sqrt(1/9) = 1/3. This is possibly all an FRM candidate might *need* to know: to increase the accuracy by a multiple of (n) implies multiplying the SE by (1/n) which generally requires the sample to multiply by n^2.

• Consequently, in 400.2, you don't even need the calcs, really, you just need to see that, ceteris paribus, the desired accuracy is a multiplier of 60/300 =1/5 which requires a scenario multiple of 5^2 = 25.
• In the next question, the SE is scaling by 1/4, from 0.8 to 0.2, which we might call "a fourfold increase in accuracy), and this will require sample (trials) multiplier of 4^2 = 16 so that SQRT(1/16) = 1/4.

I hope that explains, thanks!

 

[Laura517]

This helps a lot! Thank you so much David!

 

[nhuxng]

Hi David,

why do we multiply SE by 1/n to increase the accuracy by a multiple of n if “SE scales with the sqrt(1/n)”? Do we (incorrectly) use n for the sample size as well as for the multiplier to increase the accuracy / reduce the CI length?

Here are my thoughts on the different approaches for question 400.2 trying to find the connection and linking them to your explanations:

My understanding of your statement here “The 1/sqrt(N) indicates the key relationship between the length of the interval and sample size” is that for any given µ, critical-t, and sample standard deviation, we can only reduce the CI length by adjusting the SE through adjusting the sample size. This is what you do in your solution ”New CI length = $300 * [1/SQRT(x/800)]”. Hence, divide the CI length by sqrt(800) to eliminate the initial scale of 1/sqrt(800) and multiply by 1/sqrt(n) and find the new sample size n = 20,000. Here, n as the sample size is in line with ”SE scales with the sqrt(1/n). This part makes sense to me.

Simply stated, the CI length is 300 and we want to find the multiplier to decrease the CI length to 60, so that we solve for the multiplier in 300*[1/sqrt(multiplier)] = 60. We find that the multiplier is equal to 25, so that the sample size increases from initially 800 to 20,000 (800*25). Hence, it makes more sense to me to state “to increase the accuracy by a multiple of x implies multiplying the SE by 1/sqrt(x^2) or 1/x which generally requires the sample to multiply by x^2” to differentiate between the sample size and the multiplier. With the example it would then correctly read as “to increase the accuracy by a multiple of 5, i.e. to decrease the CI length from 300 to 60, implies multiplying the SE by 1/5 which generally requires the sample to multiply by 5^2, i.e. by 25”, so that we multiply the initial sample of 800 by 25.

Could you please clarify?

Best,

 

[David Harper CFA FRM]

Hi @nhuxng Yes, that's excellent, thank you! I agree that I'm probably (above) mixing the sample size with the multiplier in my text; e.g., "... in each of our relevant cases, the standard error scales by 1/SQRT(n), where (n) is the multiplier on the sample size (or number of trials, in MCS)." I fully agree with your ...

Hence, it makes more sense to me to state “to increase the accuracy by a multiple of x implies multiplying the SE by 1/sqrt(x^2) or 1/x which generally requires the sample to multiply by x^2” to differentiate between the sample size and the multiplier.

To maximize intuition, let me define (m) as the sample size multiplier such that n = current sample size, while m*n = updated sample sample size. So we are saying (to "steal" from you): given current SE, to achieve an updated (better) standard error given by SE/m = SE*1/m, we should increase sample size to n*m^2, or equivalently, we should scale the sample by m^2. I think (more by luck than anticipation) that I perhaps side-stepped the ambiguity with a terse solution to 400.2:

400.2. D. 20,000. New CI length = $300 * [1/SQRT(x/800)], such that x = 25*800 = 20,000; i.e., 25-fold increase in trials in order to achieve 1/SQRT(25) = 1/5th the interval.

... I don't think the given solution requires a correction (do you disagree?) although perhaps it is not as intuitive as something like the following:

"Where m is a multiplier, to improve (by decreasing) the SE to SE*1/m, we require an increase in the sample size from n to n*m^2. In this case, where Peter's manager seeks an improvement in the 95.0% CI length from $300 to $60, where CI length = 2*Margin_error = 2*Z(α)*SE = $300; she seeks $300*(m = 1/5) = $60 and therefore needs to scale (multiply) the sample by m^2 = 5^2 = 25 which implies a new sample size of 800*25 = 20,000."

I suppose the sigma can be inferred, come to think of it, although it was not my intention (the square root relationship is general and does not require the specifics). As the question does specify 95.0% confidence and, I think (assumption) per CLT that normality applies, we can infer the original sigma:

CI length = 2*MOE = 2*Z(α)*SE =2*1.645*[σ/SQRT(800)] = $300 -->
σ = 300*SQRT(800)/(2*1.645) = $2,579; i.e., where original SE is σ/SQRT(800) = $2,579/sqrt(800) = $91.19.

Then scaling (multiplying) the sample size by 5^2 leads to:
CI length = 2*Z(α)*SE =2*1.645*[2,579/SQRT(20,000)] = 2*1.645*$18.24 = $60; i.e., revised SE = $2,579/sqrt(20,000) = $18.24

Of course the giant unrealistic ceteris paribus (assumption) is that the sample variance doesn't change when the sample size increases. Actually, nevermind that: the question implicitly assumes the original sample variance is an unbiased estimate such that it can presuppose the larger-sample variance has the same expected value, i think. Thank you again!