--- title: NumPy Evaluation Metrics keywords: fastai sidebar: home_sidebar summary: "Metrics for evaluation." description: "Metrics for evaluation." nb_path: "nbs/losses__numpy.ipynb" ---
{% raw %}
{% endraw %} {% raw %}
{% endraw %} {% raw %}

divide_no_nan[source]

divide_no_nan(a, b)

Auxiliary funtion to handle divide by 0

{% endraw %} {% raw %}
{% endraw %} {% raw %}

metric_protections[source]

metric_protections(y:ndarray, y_hat:ndarray, weights:ndarray)

{% endraw %} {% raw %}
{% endraw %} {% raw %}

mape[source]

mape(y:ndarray, y_hat:ndarray, weights:Optional[ndarray]=None, axis:Optional[int]=None)

Calculates Mean Absolute Percentage Error.

MAPE measures the relative prediction accuracy of a forecasting method by calculating the percentual deviation of the prediction and the true value at a given time and averages these devations over the length of the series.

Parameters

y: numpy array Actual test values. y_hat: numpy array Predicted values. weights: numpy array, optional Weights for weighted average. axis: None or int, optional Axis or axes along which to average a. The default, axis=None, will average over all of the elements of the input array. If axis is negative it counts from the last to the first axis.

Returns

mape: numpy array or double Return the mape along the specified axis.

{% endraw %} {% raw %}
{% endraw %} {% raw %}

mse[source]

mse(y:ndarray, y_hat:ndarray, weights:Optional[ndarray]=None, axis:Optional[int]=None)

Calculates Mean Squared Error.

MSE measures the prediction accuracy of a forecasting method by calculating the squared deviation of the prediction and the true value at a given time and averages these devations over the length of the series.

Parameters

y: numpy array Actual test values. y_hat: numpy array Predicted values. weights: numpy array Weights for weighted average. axis: None or int, optional Axis or axes along which to average a. The default, axis=None, will average over all of the elements of the input array. If axis is negative it counts from the last to the first axis.

Returns

mse: numpy array or double Return the mse along the specified axis.

{% endraw %} {% raw %}
{% endraw %} {% raw %}

rmse[source]

rmse(y:ndarray, y_hat:ndarray, weights:Optional[ndarray]=None, axis:Optional[int]=None)

Calculates Root Mean Squared Error.

RMSE measures the prediction accuracy of a forecasting method by calculating the squared deviation of the prediction and the true value at a given time and averages these devations over the length of the series. Finally the RMSE will be in the same scale as the original time series so its comparison with other series is possible only if they share a common scale.

Parameters

y: numpy array Actual test values. y_hat: numpy array Predicted values. weights: numpy array Weights for weighted average. axis: None or int, optional Axis or axes along which to average a. The default, axis=None, will average over all of the elements of the input array. If axis is negative it counts from the last to the first axis.

Returns

rmse: numpy array or double Return the rmse along the specified axis.

{% endraw %} {% raw %}
{% endraw %} {% raw %}

smape[source]

smape(y:ndarray, y_hat:ndarray, weights:Optional[ndarray]=None, axis:Optional[int]=None)

Calculates Symmetric Mean Absolute Percentage Error.

SMAPE measures the relative prediction accuracy of a forecasting method by calculating the relative deviation of the prediction and the true value scaled by the sum of the absolute values for the prediction and true value at a given time, then averages these devations over the length of the series. This allows the SMAPE to have bounds between 0% and 200% which is desireble compared to normal MAPE that may be undetermined.

Parameters

y: numpy array Actual test values. y_hat: numpy array Predicted values. weights: numpy array Weights for weighted average. axis: None or int, optional Axis or axes along which to average a. The default, axis=None, will average over all of the elements of the input array. If axis is negative it counts from the last to the first axis.

Returns

smape: numpy array or double Return the smape along the specified axis.

{% endraw %} {% raw %}
{% endraw %} {% raw %}

mase[source]

mase(y:ndarray, y_hat:ndarray, y_train:ndarray, seasonality:int, weights:Optional[ndarray]=None, axis:Optional[int]=None)

Calculates the Mean Absolute Scaled Error.

MASE measures the relative prediction accuracy of a forecasting method by comparinng the mean absolute errors of the prediction and the true value against the mean absolute errors of the seasonal naive model.

Parameters

y: numpy array Actual test values. y_hat: numpy array Predicted values. y_train: numpy array Actual insample values for Seasonal Naive predictions. seasonality: int Main frequency of the time series Hourly 24, Daily 7, Weekly 52, Monthly 12, Quarterly 4, Yearly 1. weights: numpy array Weights for weighted average. axis: None or int, optional Axis or axes along which to average a. The default, axis=None, will average over all of the elements of the input array. If axis is negative it counts from the last to the first axis.

Returns

mase: numpy array or double Return the mase along the specified axis.

References

[1] https://robjhyndman.com/papers/mase.pdf

{% endraw %} {% raw %}
{% endraw %} {% raw %}

mae[source]

mae(y:ndarray, y_hat:ndarray, weights:Optional[ndarray]=None, axis:Optional[int]=None)

Calculates Mean Absolute Error.

The mean absolute error

Parameters

y: numpy array Actual test values. y_hat: numpy array Predicted values. weights: numpy array Weights for weighted average. axis: None or int, optional Axis or axes along which to average a. The default, axis=None, will average over all of the elements of the input array. If axis is negative it counts from the last to the first axis.

Returns

mae: numpy array or double Return the mae along the specified axis.

{% endraw %} {% raw %}
{% endraw %} {% raw %}

pinball_loss[source]

pinball_loss(y:ndarray, y_hat:ndarray, tau:float=0.5, weights:Optional[ndarray]=None, axis:Optional[int]=None)

Calculates the Pinball Loss.

The Pinball loss measures the deviation of a quantile forecast. By weighting the absolute deviation in a non symmetric way, the loss pays more attention to under or over estimation. A common value for tau is 0.5 for the deviation from the median.

Parameters

y: numpy array Actual test values. y_hat: numpy array Predicted values. weights: numpy array Weights for weighted average. tau: float Fixes the quantile against which the predictions are compared. axis: None or int, optional Axis or axes along which to average a. The default, axis=None, will average over all of the elements of the input array. If axis is negative it counts from the last to the first axis.

Returns

pinball loss: numpy array or double Return the pinball loss along the specified axis.

{% endraw %} {% raw %}
{% endraw %} {% raw %}

rmae[source]

rmae(y:ndarray, y_hat1:ndarray, y_hat2:ndarray, weights:Optional[ndarray]=None, axis:Optional[int]=None)

Calculates Relative Mean Absolute Error.

The relative mean absolute error of two forecasts. A number smaller than one implies that the forecast in the numerator is better than the forecast in the denominator.

Parameters

y: numpy array Actual test values. y_hat1: numpy array Predicted values of first model. y_hat2: numpy array Predicted values of second model. weights: numpy array Weights for weighted average. axis: None or int, optional Axis or axes along which to average a. The default, axis=None, will average over all of the elements of the input array. If axis is negative it counts from the last to the first axis.

Returns

rmae: numpy array or double Return the rmae along the specified axis.

{% endraw %} {% raw %}
{% endraw %}

Multi-quantile NumPy loss

MQLoss definition and testing.

{% raw %}

mqloss[source]

mqloss(y:ndarray, y_hat:ndarray, quantiles:ndarray, weights:Optional[ndarray]=None, axis:Optional[int]=None)

Calculates the MultiQuantile loss.

Calculates Average Multi-quantile Loss function, for a given set of quantiles, based on the absolute difference between predicted and true values.

Parameters

y: numpy array Actual test values. y_hat: numpy array (-1, n_quantiles) Predicted values. quantiles: numpy array (n_quantiles) Quantiles to estimate from the distribution of y. weights: numpy array Weights for weighted average. axis: None or int, optional Axis or axes along which to average a. The default, axis=None, will average over all of the elements of the input array. If axis is negative it counts from the last to the first axis.

Returns

mqloss: numpy array or double Return the mqloss along the specified axis.

{% endraw %} {% raw %}
{% endraw %} {% raw %}
y = np.random.random(size=(100, 7))
y_q = np.random.random(size=(100, 7, 4))
weights = np.ones_like(y_q)
quantiles = np.array([0.1, 0.2, 0.3, 0.4])

mqloss(y, y_q, quantiles, weights=weights, axis=(1, 2))
array([0.21041164, 0.15391882, 0.09449471, 0.18244145, 0.16071395,
       0.13142336, 0.20231349, 0.19795971, 0.15299534, 0.16744432,
       0.13345362, 0.16700624, 0.10257573, 0.16436093, 0.16341954,
       0.15676824, 0.20708862, 0.15711264, 0.21645739, 0.17206218,
       0.23661543, 0.16738924, 0.1577535 , 0.16585761, 0.16793635,
       0.15740434, 0.13467679, 0.21414516, 0.14007157, 0.27435823,
       0.20451858, 0.11715621, 0.12763097, 0.2100278 , 0.10294569,
       0.16492288, 0.11550836, 0.19200599, 0.12251248, 0.17777111,
       0.20632066, 0.25534812, 0.22850697, 0.2117134 , 0.20114755,
       0.22731604, 0.15800838, 0.17286858, 0.14366458, 0.14034129,
       0.20393408, 0.13572314, 0.24699116, 0.17890431, 0.11960552,
       0.11549357, 0.13940521, 0.13799181, 0.21941735, 0.10874776,
       0.16519921, 0.12306689, 0.1858083 , 0.11227548, 0.14478089,
       0.23283138, 0.16608539, 0.2198971 , 0.23463294, 0.22970342,
       0.16031605, 0.16675414, 0.17982392, 0.13855352, 0.22509618,
       0.18950097, 0.2207443 , 0.15073855, 0.19502465, 0.16299692,
       0.11750915, 0.17005201, 0.17436329, 0.17390164, 0.14895287,
       0.14644051, 0.10446581, 0.13071105, 0.13252147, 0.10070003,
       0.1649476 , 0.28120334, 0.13300598, 0.19693479, 0.16165529,
       0.184198  , 0.19409348, 0.14772852, 0.11654125, 0.16790548])
{% endraw %} {% raw %}
mqloss(y, y_q, quantiles)
0.16948741448706903
{% endraw %}

Checks for NumPy Evaluation Metrics

{% raw %}
y = np.array([1,1,1,0,0,0,0,0,1, np.nan])
y_mask = np.array([1,1,1,1,1,1,1,1,2,0])
y_hat = np.array([1,2,3,-4,-5,-6,-7,-8,-9,-10])

print(mae(y=y, y_hat=y_hat, weights=y_mask))
print(mae(y=y, y_hat=y_hat))
5.3
4.777777777777778
{% endraw %} {% raw %}
print(mae(y=y, y_hat=y_hat, weights=y_mask))
print(mae(y=y, y_hat=y_hat))
5.3
4.777777777777778
{% endraw %} {% raw %}
len(y)
10
{% endraw %}