Royal Bank of Scotland says the way it deals with customers' complaints is to be investigated by the financial watchdog.