The Financial Services Authority has handed down its largest ever penalty.

The official City watchdog fined insurer Royal Scottish Assurance £2 million for mis-selling and miscalculating its endowment mortgages in the early 1990s.

Up to 30,000 customers could be affected and the RSA has had to set aside £50 million to cover claims for compensation.

The fine is a record for the FSA, and matches the largest ever by a City regulator, against investment bank Deutsche Morgan Grenfell in 1997 by IMRO, the Investment Management Regulatory Organisation.

The fine is unlikely to be the last against a financial institution, as the FSA said it was in "detailed discussions" with a number of firms about endowment mis-selling.

But the number of homeowners likely to be affected would be thousands rather than the millions once expected to have lost out, it said.

Endowment mortgages are mortgages where the borrower takes out an insurance policy to pay off the loan.

While, at their peak in 1988, endowment mortgages accounted for more than 80 per cent of all new mortgages, today the investment background is very different from the high inflation, high interest rate climate of that time.

This means some endowment policies now look unlikely to produce a big enough sum to pay off the mortgage they were intended to clear.

In the Royal Scottish Assurance case, the FSA criticised the life insurance firm for "serious deficiencies" in the way it handled, marketed and calculated its Flexible Mortgage Plan endowment policy between 1990 and 1994.

The authority said RSA "failed to act with due skill, care and diligence" in calculating the premiums for its plan.

Premiums were set at such a level that policies would not repay the outstanding mortgage debt when it matured.

When selling plans, RSA said it was a provider of endowments complementary to the mortgage loans provided by Royal Bank of Scotland - one half of the joint venture with Scottish Equitable that created RSA in October 1990.

"RSA implied a level of expertise that was absent as was evidenced by its incorrect pricing of its endowment policy premiums," said the FSA.