Home » Analyses » Performance of Bondora Rating in 2014 (simplified)
          

Performance of Bondora Rating in 2014 (simplified)

This is the simplified version of the results and conclusions I did on the Bondora Rating for loans issued in 2014. A more thorough post with explanations about what I did and what I looked at in different steps of the analyses is available here. If you understand data analysis and some statistics, you should prefer to read that one. It covers the same content, but in a more detailed way and gives you information on how to replicate the analysis if you’d want to.

If you’re not that good at it, I still suggest, that you at least review this analysis version afterwards and make sure that my methods and calculations seem to make sense for you, because:

a) it’s not too smart to blindly believe any numbers you see without even looking at how these have been reached and
b) it’s always good to learn a little about data analysis and statistics, when you invest in such places like P2P-/marketplace lending, otherwise you can easily make wrong decisions with your investments and won’t understand the meaning of stats and figures shown to you.

I recently analyzed the performance and accuracy of Bondora Rating on first 6 months of loans issued and priced by that model in 2015. However, since some of you don’t understand Estonian and Google Translate is not very helpful, then I was asked to translate it into English.

Translating is relatively boring and takes a lot of time, so I won’t be translating it. You can grab my analysis file and dataset for yourself (download link below) if you’re interested and I will add a short summary of the results here below:

Bondora ESP expected loss for loans issued Jan-Jun 2015

Image 1: Expected Loss according to Bondora Rating compared to actual Expected Loss as of 21.11.2015 for ESP loans issued Jan-Jun 2015, when assuming default rate as of 21.11.2015. and same recovery as used in Bondora Rating.

Bondora FIN expected loss for loans issued Jan-Jun 2015

Image 2: Expected Loss according to Bondora Rating compared to actual Expected Loss as of 21.11.2015 for FIN loans issued Jan-Jun 2015, when assuming default rate as of 21.11.2015. and same recovery as used in Bondora Rating.

In conclusion, the Bondora Rating has been relatively inaccurate so far for several Ratings for the non-Estonian loans issued and priced within the first 6 months of 2015. Keep in mind though that some of the segments only have a small number of loans and thus should only be used to gauge the possible outcome of the investment, not the accuracy of Bondora Rating itself yet, since the result could well be random (see Tabel 3 from my analysis for the number of loans in each segment).

There was only 1 Slovakian loan issued during that time period and Estonian loans looked ok, as in line with what the Ratings would expect as of 21.11.2015., even though we don’t know the actual expected recovery for those to see more accurate results.

So, instead of translating the Estonian analysis, I’ve decided to do a similar analysis on loans with Bondora Rating that were issued in 2014 and mostly had Ratings calculated and assigned after they were issued.

I will attempt to recreate this analysis done by Bondora in May, where they took a look at the performance of different loan categories to see if the Expected Loss rates used in pricing are in line with the actual default levels, and compare the results with mine, look into more details as in split up the Ratings and countries and see what the situation is like today.

While Bondora’s analysis didn’t mention this part, it is good to keep in mind that for these loans that received a Rating after they were issued, Bondora Rating may not be entirely accurate due to incomplete data at the time the Ratings were calculated (this applies for both, positive and negative results).

The Analysis and Results (simplified version)

If you are more familiar with statistics, data analysis, excel and such, I urge you to download the dataset from the link below the post to test and try to verify my results and read the post as the version where I described in detail the analysis process.

This is the section with a shorter summary of my findings and the process for those who find it difficult to navigate through the longer explanations and descriptions of the processes.

Step 1 – Cleaning up the data

I cleared the dataset of all the irrelevant loans so that only the loans issued in 2014. were left. In other words, I only kept the data that was used by Bondora’s analysis.

Step 2 – Recreating Bondora’s analysis

First step was to recreate the Default Rate calculations for three groups of loans as of Q1 2015: AA-B, C-E and F-HR.

default rate based on Bondora Rating

Table 1: Default Rate based on loans issued in 2014.

While I expected to end up with similar result as Bondora’s analysis, my outcome was somewhat different.

AA-B in blog seems to have peaked at around 3.5%, with my result at 3.85%.

C-E in blog is closing in on 10% and my outcome was 11.68%.

F-HR in blog is below 35%, while mine is closing to 39%.

More unexpected was that I also couldn’t make the Weighted Expected Loss figures match those in the Bondora’s analysis either, although I was using the same data of loans issued in 2014 only.

Bondora Rating Expected Loss per group

My results turned out to be a little more pessimistic, as in the default rate was higher and expected loss rate was lower in several cases than those in Bondora’s blog. If anyone has any idea, why that would be the case, let me know in the comments below.

However, there’s nothing here that would indicate whether the Rating is performing as expected or not in these results, so let’s go in a little bit further.

Step 3 – Opening up the Ratings and Countries

If we want to analyze anything about the accuracy of Bondora Rating, then we would need to at minimum split the results up per country and possibly look into the results of each Rating separately as well.

Why? Because the model should be using different rules for estimating the risk of loans from different countries and if we are putting together different countries or several Ratings, we could have two groups that are equally inaccurate, but in different directions and get a result that says everything is in order.

A simplified example would be a situation where one weather forecast (Rating F) says that it’s +20 C outside, and another one (Rating HR) says it’s +10 C, but it’s actually +15 C. If you were to only look at the average of those predictions (F-HR), you’d come up with +15 C and assume the weather forecasts were accurate.

Once I calculated the default rates for different Ratings, I also used the known Loss Given Default (LGD) to calculate the Expected Loss (EL%) as of Q1 2015 with these default rates.

If my calculated EL% (as of Q1 2015) is higher than Weighted Average EL, then the outcome of these loans as of Q1 2015 has been more negative than was expected by the Bondora Rating algorithm in the Expected Loss figure (see Table 4).

Please note however that the results for rows with very few loans may not be statistically significant and thus should not be used for judging the accuracy of the Rating calculations.

Bondora Rating Expected Loss vs Actual Loss as of Q1 2015

Table 4: Expected Loss vs Actual Expected Loss per country and Rating as of Q1 2015. Actual EL% shows what the EL% would be with Q1 2015 default rates and the recovery rate used in Bondora Rating calculations.

As of Q1 2015, the results already indicated that for several Ratings in non-Estonian markets, the default rates were higher than those expected by Bondora Rating’s Expected Loss.

Step 4 – Situation as of 26.11.2015.

While the initial default rate for 2014 loans can’t go down, it can go up as more loans mature and additional defaults come in.

For this reason, let’s go one step further from the Bondora’s analysis and look at the outcome as of 26.11.2015, when we also account for the defaults that have happened after the Q1 2015.

I have skipped Estonian loans from further analysis, since Bondora has not provided the LGD figure for those and I don’t want to show a more pessimistic outcome for them than necessary, but they seemed relatively in line with their expectations as of 26.11.2015 as well, with a more pessimistic LGD.

Bondora Rating is inaccurate for every Rating in every country besides Estonia

Table 5: Expected Loss vs Actual Expected Loss per country and Rating as of 26.11.2015 . Actual EL% shows what the EL% would be with the default rates as of 26.11.2015 and the recovery rate used in Bondora Rating algorithm.

The table highlights clearly that the default rate has gone up significantly during the roughly 8 month period after Q1 2015. In total, the change was from around 1500 defaulted loans to about 2500 as of 26.11.2015. (that’s €2.67 million additional Exposure at Default amount or a 55.8% increase in 8 months total).

Again, keep in mind that the results for rows with very few loans may not be statistically significant and thus should only be used for looking at the possible investment outcomes, not for judging the accuracy of the Rating calculations itself.

However, the results indicate that the Bondora Rating for non-Estonian loans issued in 2014. has been overly optimistic in its estimates of default rates in most cases. This means that investors have a higher default rate in the non-Estonian loan segments than expected.

Step 5 – What are the Expected Returns?

Now, default rate alone doesn’t mean much and investors could earn money even when the Bondora Rating has been overly optimistic in its expectations, albeit less than they would have expected at that risk level.

Bigger problems arise though when the miss is large enough to eat up a large chunk of the expected return or turn that return into negative return instead.

So, naturally we want to look at a possible Expected Return as well. Here we’ll use the formula used in Bondora Rating for Expected Return to assist us in finding this out:

E(R) = I – EL%

I – Interest Rate
EL% – Expected Loss
E(R) – Expected Return

The longer process for reaching the Calculated Interest Rate and Calculated Expected Return that the Bondora Rating algorithm would have assigned these loans today, is explained in the longer version of this post.

But in short, I used the Weighted EL% and today’s Expected Return figures (as of 27.11.2015) to calculate the interest rate. This enables us to calculate what the Expected Return “would be” if the loans had been priced according to their risk as well, not just rated after they were issued.

I have then used the formula above to calculate Expected Return for these loans with EL% as of 26.11.2015 in case of both, the calculated interest rate and the actual interest rate that the loans were issued with.

E(R) = I * (100% – tax_rate) – EL%

Additionally, I also calculated theses returns before taxes and after taxes, assuming the Estonian 20% income tax rate (feel free to grab the excel file from below and run calculations with your tax rate if it’s different), that doesn’t allow to discount any taken losses.

Bondora Rating Returns 2014 Spain, Finland, Slovakia

Table 9: Returns according to Bondora Rating calculation with estimated and actual interest rates, before and after taxes.

The results indicate that if the loans had also been priced at the time when Ratings were calculated, according to Bondora Rating, and your tax rate was 0%, then Spanish and Finnish HR loans would have the highest E(R) as of 26.11.2015. After taxes however, the return is mostly lost to taxes and the risk level is a lot higher than this return would warrant.

With the actual interest rates that these loans were issued with however (prior to risk-based pricing), the results indicate that lower risk Finnish loans are likely to end up with a better outcome and the rest of the non-Estonian loan groups show a negative Expected Return before and after taxes.

Estonian loans were the best performers of the loans issued in 2014.

Conclusions

The Ratings given to 2014. loans, seem most accurate for Estonian loans. For non-Estonian loans, the results indicate that the model has been too optimistic about default rates and actual results are already worse as of 26.11.2015.

Actual default rates are also higher than expected with Bondora Rating for the non-Estonian segments with lower amount of loans, although that information is only relevant when looking at the possible outcome of this investment for the investor, and should not be used for gauging the accuracy of the Bondora Rating in these segments itself.

With the actual interest rates, the Expected Returns on this group of €9.3 million worth of non-Estonian loans, are in majority negative as of 26.11.2015, with or without taxes.

There is of course always the possibility that recovery is better than the expected 10% for these loans, but even then, for certain groups it would have to be a whole lot better than that to reach break-even and cover the collection expenses. This would help the investor, but wouldn’t increase the accuracy of the Bondora Rating estimations for these loans.

Download the datasets here:

Subscribe to not miss any future updates

5 Comments

  1. Comment by CoolRunning:

    A very valuable analysis

    Thanks !

    CoolRunning

  2. Comment by Mart R:

    This is ground breaking reserch work. Keep up the good work. Its going to be intresting to see Bondoras counter analysis about it..

  3. Ping from Bondora Rating has outperformed expectations? - RahaFoorum:

    […] » Analyses » Bondora Rating has outperformed expectations? « Performance of Bondora Rating in 2014 (simplified) […]

  4. Ping from Is Bondora giving away your money? - RahaFoorum:

    […] There’s one issue with that table though. Since it includes all loans, then things might have changed by quite a bit. Especially when we consider that risk-based pricing was introduced in 2015 and the non-EST loans from 2014 were pretty disastrous in several segments. […]

  5. Ping from Stats according to Bondora - RahaFoorum:

    […] some 2014 loans (inspired by Bondora’s own similar earlier analysis) and concluded that the Bondora Rating applied on those loans had significantly misjudged the expected loss for those […]

What do you think?