Hide keyboard shortcuts

Hot-keys on this page

r m x p   toggle line displays

j k   next/prev highlighted chunk

0   (zero) top of page

1   (one) first highlighted chunk

1# -*- coding: utf-8 -*- 

2"""some measures for evaluation of prediction, tests and model selection 

3 

4Created on Tue Nov 08 15:23:20 2011 

5 

6Author: Josef Perktold 

7License: BSD-3 

8 

9""" 

10import numpy as np 

11 

12from statsmodels.tools.validation import array_like 

13 

14 

15def mse(x1, x2, axis=0): 

16 """mean squared error 

17 

18 Parameters 

19 ---------- 

20 x1, x2 : array_like 

21 The performance measure depends on the difference between these two 

22 arrays. 

23 axis : int 

24 axis along which the summary statistic is calculated 

25 

26 Returns 

27 ------- 

28 mse : ndarray or float 

29 mean squared error along given axis. 

30 

31 Notes 

32 ----- 

33 If ``x1`` and ``x2`` have different shapes, then they need to broadcast. 

34 This uses ``numpy.asanyarray`` to convert the input. Whether this is the 

35 desired result or not depends on the array subclass, for example 

36 numpy matrices will silently produce an incorrect result. 

37 """ 

38 x1 = np.asanyarray(x1) 

39 x2 = np.asanyarray(x2) 

40 return np.mean((x1-x2)**2, axis=axis) 

41 

42 

43def rmse(x1, x2, axis=0): 

44 """root mean squared error 

45 

46 Parameters 

47 ---------- 

48 x1, x2 : array_like 

49 The performance measure depends on the difference between these two 

50 arrays. 

51 axis : int 

52 axis along which the summary statistic is calculated 

53 

54 Returns 

55 ------- 

56 rmse : ndarray or float 

57 root mean squared error along given axis. 

58 

59 Notes 

60 ----- 

61 If ``x1`` and ``x2`` have different shapes, then they need to broadcast. 

62 This uses ``numpy.asanyarray`` to convert the input. Whether this is the 

63 desired result or not depends on the array subclass, for example 

64 numpy matrices will silently produce an incorrect result. 

65 """ 

66 x1 = np.asanyarray(x1) 

67 x2 = np.asanyarray(x2) 

68 return np.sqrt(mse(x1, x2, axis=axis)) 

69 

70 

71def maxabs(x1, x2, axis=0): 

72 """maximum absolute error 

73 

74 Parameters 

75 ---------- 

76 x1, x2 : array_like 

77 The performance measure depends on the difference between these two 

78 arrays. 

79 axis : int 

80 axis along which the summary statistic is calculated 

81 

82 Returns 

83 ------- 

84 maxabs : ndarray or float 

85 maximum absolute difference along given axis. 

86 

87 Notes 

88 ----- 

89 If ``x1`` and ``x2`` have different shapes, then they need to broadcast. 

90 This uses ``numpy.asanyarray`` to convert the input. Whether this is the 

91 desired result or not depends on the array subclass. 

92 """ 

93 x1 = np.asanyarray(x1) 

94 x2 = np.asanyarray(x2) 

95 return np.max(np.abs(x1-x2), axis=axis) 

96 

97 

98def meanabs(x1, x2, axis=0): 

99 """mean absolute error 

100 

101 Parameters 

102 ---------- 

103 x1, x2 : array_like 

104 The performance measure depends on the difference between these two 

105 arrays. 

106 axis : int 

107 axis along which the summary statistic is calculated 

108 

109 Returns 

110 ------- 

111 meanabs : ndarray or float 

112 mean absolute difference along given axis. 

113 

114 Notes 

115 ----- 

116 If ``x1`` and ``x2`` have different shapes, then they need to broadcast. 

117 This uses ``numpy.asanyarray`` to convert the input. Whether this is the 

118 desired result or not depends on the array subclass. 

119 """ 

120 x1 = np.asanyarray(x1) 

121 x2 = np.asanyarray(x2) 

122 return np.mean(np.abs(x1-x2), axis=axis) 

123 

124 

125def medianabs(x1, x2, axis=0): 

126 """median absolute error 

127 

128 Parameters 

129 ---------- 

130 x1, x2 : array_like 

131 The performance measure depends on the difference between these two 

132 arrays. 

133 axis : int 

134 axis along which the summary statistic is calculated 

135 

136 Returns 

137 ------- 

138 medianabs : ndarray or float 

139 median absolute difference along given axis. 

140 

141 Notes 

142 ----- 

143 If ``x1`` and ``x2`` have different shapes, then they need to broadcast. 

144 This uses ``numpy.asanyarray`` to convert the input. Whether this is the 

145 desired result or not depends on the array subclass. 

146 """ 

147 x1 = np.asanyarray(x1) 

148 x2 = np.asanyarray(x2) 

149 return np.median(np.abs(x1-x2), axis=axis) 

150 

151 

152def bias(x1, x2, axis=0): 

153 """bias, mean error 

154 

155 Parameters 

156 ---------- 

157 x1, x2 : array_like 

158 The performance measure depends on the difference between these two 

159 arrays. 

160 axis : int 

161 axis along which the summary statistic is calculated 

162 

163 Returns 

164 ------- 

165 bias : ndarray or float 

166 bias, or mean difference along given axis. 

167 

168 Notes 

169 ----- 

170 If ``x1`` and ``x2`` have different shapes, then they need to broadcast. 

171 This uses ``numpy.asanyarray`` to convert the input. Whether this is the 

172 desired result or not depends on the array subclass. 

173 """ 

174 x1 = np.asanyarray(x1) 

175 x2 = np.asanyarray(x2) 

176 return np.mean(x1-x2, axis=axis) 

177 

178 

179def medianbias(x1, x2, axis=0): 

180 """median bias, median error 

181 

182 Parameters 

183 ---------- 

184 x1, x2 : array_like 

185 The performance measure depends on the difference between these two 

186 arrays. 

187 axis : int 

188 axis along which the summary statistic is calculated 

189 

190 Returns 

191 ------- 

192 medianbias : ndarray or float 

193 median bias, or median difference along given axis. 

194 

195 Notes 

196 ----- 

197 If ``x1`` and ``x2`` have different shapes, then they need to broadcast. 

198 This uses ``numpy.asanyarray`` to convert the input. Whether this is the 

199 desired result or not depends on the array subclass. 

200 """ 

201 x1 = np.asanyarray(x1) 

202 x2 = np.asanyarray(x2) 

203 return np.median(x1-x2, axis=axis) 

204 

205 

206def vare(x1, x2, ddof=0, axis=0): 

207 """variance of error 

208 

209 Parameters 

210 ---------- 

211 x1, x2 : array_like 

212 The performance measure depends on the difference between these two 

213 arrays. 

214 axis : int 

215 axis along which the summary statistic is calculated 

216 

217 Returns 

218 ------- 

219 vare : ndarray or float 

220 variance of difference along given axis. 

221 

222 Notes 

223 ----- 

224 If ``x1`` and ``x2`` have different shapes, then they need to broadcast. 

225 This uses ``numpy.asanyarray`` to convert the input. Whether this is the 

226 desired result or not depends on the array subclass. 

227 """ 

228 x1 = np.asanyarray(x1) 

229 x2 = np.asanyarray(x2) 

230 return np.var(x1-x2, ddof=ddof, axis=axis) 

231 

232 

233def stde(x1, x2, ddof=0, axis=0): 

234 """standard deviation of error 

235 

236 Parameters 

237 ---------- 

238 x1, x2 : array_like 

239 The performance measure depends on the difference between these two 

240 arrays. 

241 axis : int 

242 axis along which the summary statistic is calculated 

243 

244 Returns 

245 ------- 

246 stde : ndarray or float 

247 standard deviation of difference along given axis. 

248 

249 Notes 

250 ----- 

251 If ``x1`` and ``x2`` have different shapes, then they need to broadcast. 

252 This uses ``numpy.asanyarray`` to convert the input. Whether this is the 

253 desired result or not depends on the array subclass. 

254 """ 

255 x1 = np.asanyarray(x1) 

256 x2 = np.asanyarray(x2) 

257 return np.std(x1-x2, ddof=ddof, axis=axis) 

258 

259 

260def iqr(x1, x2, axis=0): 

261 """ 

262 Interquartile range of error 

263 

264 Parameters 

265 ---------- 

266 x1 : array_like 

267 One of the inputs into the IQR calculation. 

268 x2 : array_like 

269 The other input into the IQR calculation. 

270 axis : {None, int} 

271 axis along which the summary statistic is calculated 

272 

273 Returns 

274 ------- 

275 irq : {float, ndarray} 

276 Interquartile range along given axis. 

277 

278 Notes 

279 ----- 

280 If ``x1`` and ``x2`` have different shapes, then they must broadcast. 

281 """ 

282 x1 = array_like(x1, 'x1', dtype=None, ndim=None) 

283 x2 = array_like(x2, 'x1', dtype=None, ndim=None) 

284 if axis is None: 

285 x1 = x1.ravel() 

286 x2 = x2.ravel() 

287 axis = 0 

288 xdiff = np.sort(x1 - x2, axis=axis) 

289 nobs = x1.shape[axis] 

290 idx = np.round((nobs-1) * np.array([0.25, 0.75])).astype(int) 

291 sl = [slice(None)] * xdiff.ndim 

292 sl[axis] = idx 

293 iqr = np.diff(xdiff[tuple(sl)], axis=axis) 

294 iqr = np.squeeze(iqr) # drop reduced dimension 

295 return iqr 

296 

297 

298# Information Criteria 

299# --------------------- 

300 

301def aic(llf, nobs, df_modelwc): 

302 """Akaike information criterion 

303 

304 Parameters 

305 ---------- 

306 llf : float 

307 value of the loglikelihood 

308 nobs : int 

309 number of observations 

310 df_modelwc : int 

311 number of parameters including constant 

312 

313 Returns 

314 ------- 

315 aic : float 

316 information criterion 

317 

318 References 

319 ---------- 

320 https://en.wikipedia.org/wiki/Akaike_information_criterion 

321 """ 

322 return -2. * llf + 2. * df_modelwc 

323 

324 

325def aicc(llf, nobs, df_modelwc): 

326 """Akaike information criterion (AIC) with small sample correction 

327 

328 Parameters 

329 ---------- 

330 llf : float 

331 value of the loglikelihood 

332 nobs : int 

333 number of observations 

334 df_modelwc : int 

335 number of parameters including constant 

336 

337 Returns 

338 ------- 

339 aicc : float 

340 information criterion 

341 

342 References 

343 ---------- 

344 https://en.wikipedia.org/wiki/Akaike_information_criterion#AICc 

345 """ 

346 return -2. * llf + 2. * df_modelwc * nobs / (nobs - df_modelwc - 1.) 

347 

348 

349def bic(llf, nobs, df_modelwc): 

350 """Bayesian information criterion (BIC) or Schwarz criterion 

351 

352 Parameters 

353 ---------- 

354 llf : float 

355 value of the loglikelihood 

356 nobs : int 

357 number of observations 

358 df_modelwc : int 

359 number of parameters including constant 

360 

361 Returns 

362 ------- 

363 bic : float 

364 information criterion 

365 

366 References 

367 ---------- 

368 https://en.wikipedia.org/wiki/Bayesian_information_criterion 

369 """ 

370 return -2. * llf + np.log(nobs) * df_modelwc 

371 

372 

373def hqic(llf, nobs, df_modelwc): 

374 """Hannan-Quinn information criterion (HQC) 

375 

376 Parameters 

377 ---------- 

378 llf : float 

379 value of the loglikelihood 

380 nobs : int 

381 number of observations 

382 df_modelwc : int 

383 number of parameters including constant 

384 

385 Returns 

386 ------- 

387 hqic : float 

388 information criterion 

389 

390 References 

391 ---------- 

392 Wikipedia does not say much 

393 """ 

394 return -2. * llf + 2 * np.log(np.log(nobs)) * df_modelwc 

395 

396 

397# IC based on residual sigma 

398 

399def aic_sigma(sigma2, nobs, df_modelwc, islog=False): 

400 r"""Akaike information criterion 

401 

402 Parameters 

403 ---------- 

404 sigma2 : float 

405 estimate of the residual variance or determinant of Sigma_hat in the 

406 multivariate case. If islog is true, then it is assumed that sigma 

407 is already log-ed, for example logdetSigma. 

408 nobs : int 

409 number of observations 

410 df_modelwc : int 

411 number of parameters including constant 

412 

413 Returns 

414 ------- 

415 aic : float 

416 information criterion 

417 

418 Notes 

419 ----- 

420 A constant has been dropped in comparison to the loglikelihood base 

421 information criteria. The information criteria should be used to compare 

422 only comparable models. 

423 

424 For example, AIC is defined in terms of the loglikelihood as 

425 

426 :math:`-2 llf + 2 k` 

427 

428 in terms of :math:`\hat{\sigma}^2` 

429 

430 :math:`log(\hat{\sigma}^2) + 2 k / n` 

431 

432 in terms of the determinant of :math:`\hat{\Sigma}` 

433 

434 :math:`log(\|\hat{\Sigma}\|) + 2 k / n` 

435 

436 Note: In our definition we do not divide by n in the log-likelihood 

437 version. 

438 

439 TODO: Latex math 

440 

441 reference for example lecture notes by Herman Bierens 

442 

443 See Also 

444 -------- 

445 

446 References 

447 ---------- 

448 https://en.wikipedia.org/wiki/Akaike_information_criterion 

449 """ 

450 if not islog: 

451 sigma2 = np.log(sigma2) 

452 return sigma2 + aic(0, nobs, df_modelwc) / nobs 

453 

454 

455def aicc_sigma(sigma2, nobs, df_modelwc, islog=False): 

456 """Akaike information criterion (AIC) with small sample correction 

457 

458 Parameters 

459 ---------- 

460 sigma2 : float 

461 estimate of the residual variance or determinant of Sigma_hat in the 

462 multivariate case. If islog is true, then it is assumed that sigma 

463 is already log-ed, for example logdetSigma. 

464 nobs : int 

465 number of observations 

466 df_modelwc : int 

467 number of parameters including constant 

468 

469 Returns 

470 ------- 

471 aicc : float 

472 information criterion 

473 

474 Notes 

475 ----- 

476 A constant has been dropped in comparison to the loglikelihood base 

477 information criteria. These should be used to compare for comparable 

478 models. 

479 

480 References 

481 ---------- 

482 https://en.wikipedia.org/wiki/Akaike_information_criterion#AICc 

483 """ 

484 if not islog: 

485 sigma2 = np.log(sigma2) 

486 return sigma2 + aicc(0, nobs, df_modelwc) / nobs 

487 

488 

489def bic_sigma(sigma2, nobs, df_modelwc, islog=False): 

490 """Bayesian information criterion (BIC) or Schwarz criterion 

491 

492 Parameters 

493 ---------- 

494 sigma2 : float 

495 estimate of the residual variance or determinant of Sigma_hat in the 

496 multivariate case. If islog is true, then it is assumed that sigma 

497 is already log-ed, for example logdetSigma. 

498 nobs : int 

499 number of observations 

500 df_modelwc : int 

501 number of parameters including constant 

502 

503 Returns 

504 ------- 

505 bic : float 

506 information criterion 

507 

508 Notes 

509 ----- 

510 A constant has been dropped in comparison to the loglikelihood base 

511 information criteria. These should be used to compare for comparable 

512 models. 

513 

514 References 

515 ---------- 

516 https://en.wikipedia.org/wiki/Bayesian_information_criterion 

517 """ 

518 if not islog: 

519 sigma2 = np.log(sigma2) 

520 return sigma2 + bic(0, nobs, df_modelwc) / nobs 

521 

522 

523def hqic_sigma(sigma2, nobs, df_modelwc, islog=False): 

524 """Hannan-Quinn information criterion (HQC) 

525 

526 Parameters 

527 ---------- 

528 sigma2 : float 

529 estimate of the residual variance or determinant of Sigma_hat in the 

530 multivariate case. If islog is true, then it is assumed that sigma 

531 is already log-ed, for example logdetSigma. 

532 nobs : int 

533 number of observations 

534 df_modelwc : int 

535 number of parameters including constant 

536 

537 Returns 

538 ------- 

539 hqic : float 

540 information criterion 

541 

542 Notes 

543 ----- 

544 A constant has been dropped in comparison to the loglikelihood base 

545 information criteria. These should be used to compare for comparable 

546 models. 

547 

548 References 

549 ---------- 

550 xxx 

551 """ 

552 if not islog: 

553 sigma2 = np.log(sigma2) 

554 return sigma2 + hqic(0, nobs, df_modelwc) / nobs 

555 

556 

557# from var_model.py, VAR only? separates neqs and k_vars per equation 

558# def fpe_sigma(): 

559# ((nobs + self.df_model) / self.df_resid) ** neqs * np.exp(ld) 

560 

561 

562__all__ = [maxabs, meanabs, medianabs, medianbias, mse, rmse, stde, vare, 

563 aic, aic_sigma, aicc, aicc_sigma, bias, bic, bic_sigma, 

564 hqic, hqic_sigma, iqr]