Performance of Bondora Rating in 2014 (analysis)

This is a more thorough post with explanations about what I did and what I looked at in different steps of the analyses. If statistics and data analysis is not your strong side, there is also a simplified version of the steps and conclusions available here. I do suggest however, that you at least review this version here afterwards and make sure that my methods and calculations seem to make sense for you, because:

a) it’s not too smart to blindly believe any numbers you see without even looking at how these have been reached and
b) it’s always good to learn a little about data analysis and statistics, when you invest in such places like P2P-/marketplace lending, otherwise you can easily make wrong decisions with your investments and won’t understand the meaning of stats and figures shown to you.

I recently analyzed the performance and accuracy of Bondora Rating on first 6 months of loans issued and priced by that model in 2015. However, since some of you don’t understand Estonian and Google Translate is not very helpful, then I was asked to translate it into English.

Translating is relatively boring and takes a lot of time, so I won’t be translating it. You can grab my analysis file and dataset for yourself (download link below) if you’re interested and I will add a short summary of the results here below:

In conclusion, the Bondora Rating has been relatively inaccurate so far for several Ratings for the non-Estonian loans issued and priced within the first 6 months of 2015. Keep in mind though that some of the segments only have a small number of loans and thus should only be used to gauge the possible outcome of the investment, not the accuracy of Bondora Rating itself yet, since the result could well be random (see Tabel 3 from my analysis for the number of loans in each segment).

There was only 1 Slovakian loan issued during that time period and Estonian loans seemed to be performing in line with what Bondora Rating would expect, even though we don’t know the actual expected recovery for those to see more accurate results.

So, instead of translating the Estonian analysis, I’ve decided to do a similar analysis on loans with Bondora Rating that were issued in 2014 and mostly had Ratings calculated and assigned after they were issued.

I will attempt to recreate this analysis done by Bondora in May, where they took a look at the performance of different loan categories to see if the Expected Loss rates used in pricing are in line with the actual default levels, and compare the results with mine, look into more details as in split up the Ratings and countries and see what the situation is like today.

While Bondora’s analysis didn’t mention this part, it is good to keep in mind that for these loans that received a Rating after they were issued, Bondora Rating may not be entirely accurate due to incomplete data at the time the Ratings were calculated (this applies for both, positive and negative results).

To help me identify any possible mistakes from my analyses, please try to recreate the analyses by cleaning up the data yourself and comparing it to my sample data. Also, you could help me by checking the data file for possible errors/mistakes (download below).

Cleaning the Data

To begin, I downloaded the loan dataset from Bondora’s data export page.

Since analysis included only loans issued in 2014, I cleaned out all the loans with LoanDate before and after 2014. I also removed from dataset all the loans without LoanDate, since this means that they weren’t even issued at any point and are irrelevant for our analyses. This left me with a total of 8713 rows of data + header row.

Secondly, it is mentioned in the post that the results are shown as of Q1 2015. To replicate this, fortunately, it is enough to just filter out defaults that have happened after Q1 2015, in other words, I should consider loans that defaulted since 1st of April 2015 as non-defaults to achieve same result.

For this, I created a new column Default_Before_Q2_2015 and assigned 1 to all rows where Default_StartDate is before 1st of April 2015 and 0 to all empty rows and rows with defaults that happened later.

The difference is relatively large, with close to 2500 defaults by today on same segment and 1500 before Q2, meaning that nearly 1000 new defaults have happened since the time-frame used in the analysis done by Bondora.

Additionally I created another field of Exposure_at_Default_Before_Q2 to make it easier to use pivot table to show only the defaulted amount for loans that defaulted before Q2.

Before each analysis, I also added filters to the pivot table to eliminate cancelled loans. Including using a filter I created for loans where ContractEndDate is not empty (meaning that loan has ended) and InterestAndPenaltiesPaid = 0 (meaning that loan was issued, it has now ended, but no interest was paid to investors).

Most of the filters besides the cancellations, were useless, since I deleted that data in previous steps, but I added them just in case anyway.

Replicating the Analysis

Default Rate

The first thing to do was to calculate the default rate for those 3 separate groups of loans based on amount as was done in Bondora’s analysis.

For this I took the Exposure_at_Default_Before_Q2 sum as defaulted amount and FundedAmount as the initial amount issued.

If I understood it correctly how Bondora’s analysis was done, then in theory, since I won’t split the loans into separate months, my total result should be lower than the 12-month figure in Bondora’s analyses.

However, I didn’t manage to replicate the results shown in the blog post since all of my calculated default rates were higher than the numbers indicated in the 12-month point for these groups.

AA-B in blog seems to have peaked at around 3.5%, with my result at 3.85%.

C-E in blog is closing in on 10% and my outcome was 11.68%.

F-HR in blog is below 35%, while mine is closing to 39%.

At first I thought perhaps, by origination, it was meant the time when loan application was started, not when loan was issued, so I did that calculation as well, but results were relatively similar to my results in Table 1.

There are some other possibilities here why my result is different that I won’t take the time today to test, since they would be misleading ways to calculate this, they wouldn’t make any sense to do it this way or I can’t test them:

• Instead of the Exposure_at_Default value, in this analysis they used EAD1 or EAD2. This would mean that the analysis was done with a lower figure than that used in the Bondora Rating and Expected Loss calculation itself, which would make no sense and would be misleading.
• Instead of the end of March, the “as of Q1 2015”, ended at some earlier time, like end of December 2014. Which would make no sense, considering that the analysis was done in May and there was full data present for Q1. It would also be misleading to say it was Q1, if it actually was as of Q4 2014.
• The data export data is during my analysis or was during Bondora’s analysis incorrect or I made some sort of mistake in my filtering somewhere. I have double checked those and everything at least seems correct. I also removed my own “Cancelled” filter, which could have made the result more positive if you didn’t remove those from the sample, but even this didn’t have this big effect. I will add the dataset and analyses file for you all to verify just in case though (download below).

If you know the reason why I couldn’t replicate the results and why the difference is there, make sure to let me know in the comments below.

Expected Loss

Next step would be to take a look at the Expected Loss for those groups. Since this is easily available in the Data Export file, it’s relatively easy to get the average EL into a pivot table.

However, since we calculated default rate by amounts, not count of loan and we want to compare that to EL, the latter should also be weighted, not just arithmetic average. So I calculated that one as well.

Again, I wasn’t able to replicate the results shown in Bondora analyses. In this case the Expected Loss for AA-B and C-E groups is higher in Bondora analyses than mine, and for F-HR, it’s lower than mine (irregardless if it’s weighted or not).

Even if I removed all the filters or added some, I still couldn’t manage to replicate the same expected loss levels as highlighted in Bondora’s analyses.

Conclusion

While I didn’t manage to replicate the results shown in Bondora blog, I did end up with a conclusion that was somewhat different from the results highlighted in their analysis.

Although, there didn’t seem to be anything above the expected limit by then with these groups, since Expected Loss also includes an expected rate of recovery, other groups besides F-HR already had a higher default rate than Expected Loss and the EL figures were mostly lower than highlighted in the blog.

If someone finds out the reason, why, let me know also.

However, let’s take the analysis a little bit further, since these figures don’t tell us actually much about the accuracy of the Bondora Rating itself.

Results per Bondora Rating and Country

If we want to check how well the Bondora Rating has done with predicting the EL for loans that were Rated after they were issued (which probably was mostly data that the model was built on as well I guess since there wasn’t much more data for FIN and ESP that time), then we want to see the results per Bondora Rating and per Country.

Mainly because summed up like that, the results could actually even each others’ inaccuracies out and seem as if everything is ok, while actually it may not be. We also know what the loss rate is for non-Estonian loans so it’s quite easy to check whether the results are still in line with the expected calculations or not.

If we look at how the Bondora Rating is calculated, we will get this formula for Expected Loss:

EL% = PD*LGD*EAD%

PD – Probability of Default, as in the likelihood of a loan defaulting at some point
LGD – Loss Given Default, as in the amount that is lost after all the recovery that is to be expected
EAD% – Exposure at Default, as in the outstanding principal amount + the accrued interest amount at the moment of default, the actual EAD amount is shown in the Exposure_at_Default field in Data Export file

If we want to compare today’s default rate with the EL% assumed in Bondora Rating pricing algorithm, then we need to calculate the EL% for those loans today based on the current default rate.

This is relatively easy, since we can simply assume that all of the PD and EAD has already happened (this is reflected in the default rate as of Q1 2015) by the time we do the analyses. Note that there could be more defaults in the future, since most of the loans are relatively fresh by Q1 2015 and haven’t yet been repaid, but we’ll assume the most optimistic scenario here.

We know LGD from Bondora Rating page where it’s stated that this is 90% for Spain, Finland and Slovakia. The actual figure for Estonia is not publicly available, but let’s assume the recovery is 27% (LGD of 73%) on average as highlighted in some previous Bondora blog posts and analyses.

I assume it’s quite pessimistic figure and in reality they use a lower LGD for Estonian loans. However, since Bondora hasn’t provided the actual figure and we don’t have anything more accurate available, let’s use this one to get some sort of baseline for Estonian loans and consider the result as somewhat  pessimistic scenario.

So, to get the actual EL% from default rate, we simply need to do the following calculation:

EL% = Default_Rate*LGD

For this analysis, I still looked at the defaults only as of Q1 2015 so it should be the same situation that was highlighted in Bondora’s analyses.

As seen from Table 4, Estonian loans are the only ones where the Expected Loss as of Q1 2015, would have still been in range for all Ratings according to the Bondora Rating algorithm even when we consider LGD as 73% (green field means that the difference is positive as in EL% in Rating was higher than actual EL% as of Q1 2015).

In every other country, the actual EL% was already higher by Q1 2015 than assumed in the Rating calculations. Clearly the worst outcome is with Slovakian loans, where the EL% and Rating seem to have not much correlation with actual results.

I have left uncolored all rows with less than 100 loans and you should keep in mind that rows with small amount of loans, can likely be random and thus should only be used to evaluate the possible outcome of these investments and not the accuracy of Bondora Rating for these segments per se.

Although, as of Q1 2015, a large portion of those loans was quite fresh so I would expect to see a lot more positive values here, even in the random rows.

In other words, if we look at the Bondora Rating results separately by Rating and country at the moment Bondora’s analyses were published, the results seem already worse than expected in some cases for countries other than Estonia.

Bondora Rating Accuracy

Since things can’t get any better in terms of default rates (they could get better for the investor in case there is significantly higher recovery than expected, but that wouldn’t affect the accuracy of Rating algorithm anymore), but they still could get worse, it makes sense to revisit this analysis again, as Bondora also planned to do when more of the 2014 loans have reached 12 month point.

Well, they have by now, so let’s continue to take a look at the results as of 26.11.2015.

NB! I skipped Estonian loans from further analysis, since Bondora hasn’t provided the actual LGD rate for them and I might be showing too pessimistic results due to this fact. As a side note, EST still looks relatively in lines of what the Rating seemed to expect even as of 26.11.2015 at the 73% LGD rate.

Unless the actual LGD is higher than 73%, which is unlikely in the light of historic recovery in Estonia, we can assume that as of 26.11.2015. the Estonian loans issued in 2014 are performing close to as expected.

As of 26.11.2015, based on loans issued in 2014 that have a Bondora Rating, there are no segments outside of Estonia, where the EL% used by Bondora Rating for pricing the loans, seems to be in line with the expectations. As can be seen from the fact that there are no rows with positive values (green cells).

For those non-Estonian loans, the EL% as of 26.11.2015, has been underestimated in the Ratings by an average of 23%.

Considering that we are looking at these loans when they have matured between 11 – 23 months and average loan duration for these loans (excluding Estonian loans) is 42 months, it is relatively safe to assume that there’ll be some additional defaults coming in the future.

Hopefully most of the defaults have already happened by now, but judging by past results, we could expect more and the inaccuracy can only grow for those loans.

Effect on Returns

Since these loans didn’t have the actual pricing by Bondora Rating and there is no Expected Return figure in the dataset and I seriously can’t be bothered to check all the loans one by one through Secondary Market or some other means, we cannot estimate exactly how much of the Return has been lost due to these inaccuracies.

However, we can calculate an estimated Interest Rate.

Calculating the Interest Rate

What we do know, is the EL% used by Bondora Rating and that the Interest Rate can be calculated as following:

I = EL% + E(R)

I – Interest Rate
EL% – Expected Loss
E(R) – Expected Return

However, considering that the Expected Return in the Rating seems to be highest for Estonian loans (view Table 4 in my previous analysis) and possibly Slovakian as well, we can relatively safely assume that the average Expected Return figure on Statistics page shows higher figure than the actual figure would be for those loans, perhaps with the exception of Slovakia.

Of course, when we see that the highest Expected Return currently (14:20 27.11.2015) on market is 21.39% with interest rate of 67.54%, we are relatively safe in assuming that it doesn’t go much above 22% and all of Slovakia is at a loss according to Bondora Rating calculations if the loans had been priced by it accordingly. Even if the actual loss is a bit lower than we will get with the following calculation where we use the Expected Return from Statistics page risk section.

Even, if we consider that you invested into all of those non-Estonian loans and they would have had a pricing based on Bondora Rating, and I didn’t make any serious mistakes in my calculations, you would be at a loss today without even doing any further calculations.

The highest possible Expected Return is somewhere around 22% (there are a few anomalies with higher ER in dataset, but definitely not even close to bring the average above it), while the average increase of EL% given by Bondora Rating by EL% as of 26.11.2015. here is above 23%.

In other words, the Expected Loss as of 26.11.2015. compared to the EL assumed by Rating algorithm, is covering the upper limit of possible E(R) that Bondora Rating has been giving for loans during its pricing.

I compared these ER figures from statistics page just in case with the ones I calculated for the analyses based on first 6 months of 2015 (it doesn’t include Slovakia though) and in almost all of the cases my assumption is correct and the average ER shown in statistics page is somewhat higher than it has been for loans issued in first 6 months of 2015 with the same Bondora Rating algorithm.

Only exception is FIN HR loans that have 0.14% higher ER in my analysis than on the statistics page. In other words, the calculated interest rate is relatively realistic compared to what the Rating has been giving to similar EL% loans in 2015.

We are assuming that SVK has similar expected return, since we don’t have any concrete data to tell us otherwise.

What we can see from data though, is that had the Rating been introduced a bit earlier, there would have been lower amount in defaults for Slovakia, since there was an interest cap introduced in the September 2014. for Slovakian loans and Rating would have had to reject the majority of the  ~30 loans that were issued after that because of too high assumed EL%.

Now that we have EL% and calculated E(R), we simply need to add them together to get the Interest Rate that Bondora Rating algorithm would have assigned these loans if the model had existed back then.

Calculating Expected Return

We now have all the data required to calculate current Expected Return with the Expected Loss as of 26.11.2015 and the calculated Interest Rate.

Expected Return calculation formula used in Bondora Rating is as follows:

E(R) = I – EL%

We now replace the EL% there, with the actual EL% we calculated above (see Table 5) to see what’s the actual Expected Return after those defaults as of 26.11.2015.

The results show that while the Rating has made too optimistic assumptions in regards to EL% for Rating groups in non-Estonian countries, then Finnish and Spanish HR loans still would have some room to earn a positive return at this point (before taxes at least).

The return would be less than expected during the pricing and this wouldn’t increase the accuracy of the model, but at least it’s a positive note. Should still revisit this after loans are even more mature to see if it stays this way or not.

This is positive news for those who have been investing into Spanish and Finnish HR loans in 2015 (if after paying taxes you’d still get a positive outcome and IF the loans in 2015 perform similarly), but let’s see what it means for the people who invested into those 1647 loans in 2014.

Returns for 2014 loans with actual Interest rates

If you actually invested into those loans, then what might interest you, is the outcome of those investments based on actual interest rate that was used back then. Remember, there was no risk-based pricing in 2014, so riskier loans received lower rates than they should have and lower risk loans received higher interest rates than they should have.

For this, we again use the same calculation from Bondora Rating of:

E(R) = I – EL%

Except, for I, we use the actual interest rate for loans issued in 2014.

As Table 8 illustrates, there’s not much to be optimistic about if you invested into such loans. There is still a possible positive outcome for FIN B and C loans, but they only amount to about 10% of all of these loans here.

Of course, unless you’re one of the people who is investing through an Estonian company where you’d have to pay no taxes, what actually matters at the end of the day, is the return after taxes. So let’s take a look at that as well.

Calculating Return After Taxes

While I don’t agree that Bondora’s formula for calculating the return is the most accurate way to do this, we’ll continue using it here as well, for the ease of use and for keeping the results consistent and comparable.

E(R) = I – EL%

To account for taxes, we’ll simply reduce the I (interest received) by tax rate like this:

E(R) = I * (100% – tax_rate) – EL%

In the table below, I’ll bring together results for the Expected Return for 4 scenarios:

1. Calculated Expected Return – the Expected Return we’d expect if we look at actual Expected Loss as of 26.11.2015 and use the estimated interest rates.
2. Calculated Return after Taxes – same as 1., but after taxes.
3. Return Before Taxes (Actual Interest Rates) – same as 1., but using the actual interest rate instead.
4. Return after Taxes (Actual Interest Rates) – same as 2., but using the actual interest rate instead

For tax rate, I am using Estonia’s 20% income tax, but you  can easily recalculate this with any rate by changing the value in the data file. Note that Estonian law doesn’t allow for deductions of losses from taxable income as of 2015.

So, if you’re investing with 0% tax rate, you would look at either the Calculated Expected Return (to get an idea on what might be result for 2015 loans IF they perform similarly, since they’ve been also rated by Bondora Rating model) or Return Before Taxes (Actual Interest Rates) columns, to get an idea what to expect from the 2014 loans.

If you’re paying taxes, then you’d look at either Calculated Return after Taxes or Return after Taxes (Actual Interest Rates) respectively.

If your tax rate is different from 20%, then you can calculate easily with that (if you want), by changing the value in the tax rate cell in my data file (download below).

The results would indicate that if you’re investing without any taxes, then:

• had you invested in 2014 into Finnish B and C loans, you would have some chance of getting a profit from those segments
• if the Rating is working similarly in 2015 as it does on 2014. loans, you might have a chance to earn a relatively nice return on Finnish and Spanish HR loans, although it may not necessarily be the best option considering the risk/return ratio and that it’s a big if, since the Rating didn’t seem to predict the results accurately in 2014 and it’s using some additional data for 2015 loans
• if you invested into anything else, you are possibly losing money

If you pay the Estonian 20% tax rate, then:

• had you invested into any of the non-Estonian Ratings in 2014, you are likely to lose money or in some cases, simply taking way too much risk to earn a relatively low return
• if the Rating is working similarly as it does on 2014. loans (and that’s also a big if of course), you are also likely to lose money or in some other cases, earning a low return which definitely doesn’t justify the risks you’re taking.

Conclusions

The Ratings given to 2014. loans, seem most accurate for Estonian loans. For non-Estonian loans, the results indicate that the model has been too optimistic about default rates and actual results are already worse as of 26.11.2015.

Please, grab the excel file, filter out the relevant segment yourself and double-check my calculations and formulas to make sure I haven’t made a mistake anywhere. Also, let me know what your results were when you did it and if you have any further thoughts and ideas on this topic.

To put this into context, these calculations on non-Estonian loans are done on €9.3 million worth of loans that were issued in 2014. That’s about 1/5 of all loans issued in Bondora since the beginning until 26.11.2015.

In other words, the segment of 20% of loans ever issued on Bondora, with the actual interest rates, has (as of 26.11.2015) in majority a negative Expected Return with or without taxes.

There is of course always the possibility that recovery is better than the expected 10% for these loans, but even then, for certain groups it would have to be a whole lot better than that to reach break-even and cover the collection expenses for investor. This would of course help the investor, but wouldn’t increase the accuracy of the Bondora Rating estimations for these loans.

A bigger elephant in the room is that the model is also accounting for unexpected loss and macro risk as well, such as during perhaps an economic downturn (?). The results today don’t show much room for further increase of default rates for a large portion of the portfolio however.

Has all of that macro risk and unexpected loss happened (in several cases even more of it) in these countries or what will be the outcome during such times for investors on this type of loans?

It would be nice to see a proper stress test done on Bondora’s loan portfolio and especially on non-Estonian loans, since their proportion has been growing to quite a lot by now.

Download the datasets here:

• 2014. loans
• 2015. first 6 months loans