Ransomware is becoming more complex, and anti-virus companies are worried they may not be able to decrypt ransomed files, according to a new report.