AppSOC Labs Report: DeepSeek with & without Azure Guardrails

In ongoing testing of widely used AI models, the AppSOC Research Labs has tested the version of the DeepSeek-R1 model available on Azure. Using the AppSOCAI Security Testing module, the tests compared failure rates with, and without applying content filters/guardrails available from Azure.

The DeepSeek-R1 model available on Azure produced similar results to versions previously tested that were downloaded directly from DeepSeek (see previous reports). Applying Azure’s content filters improved performance in some categories, but the model still had very high failure rates in critical areas.

✓ Azure filters significantly reduced Jailbreak failure rates for DeepSeek-R1 (5% vs. 37.6%).

✓ Toxicity failure rates were reduced more than 70% with Azure filters applied (4% vs. 14.8%).

✓ Training Data Leak failures were also reduced more than 2/3 by the Azure filters (10% vs 32.7%).

✓ Hallucination failures were eliminated by the Azure filters in the tests conducted.

✷ Supply Chain failure rates were slightly raised with the Azure filters (6.9% vs. 5.8%) but acceptable.

✘ Prompt Injection failures were reduced by Azure filters but remain very high (57.1% vs. 40%).

✘ Malware failure rates were slightly improved by Azure filters but were still dangerously high (93.8% vs. 96.7%).

✘ Virus failure rates were unchanged by Azure filters remaining unacceptably high (93.3%).

✘ Overall, AppSOC Knowledge Base risk scores rate DeepSeek-R1 with filters at 8.3/10, and DeepSeek-R1 without filters at 8.4/10. Both are rated as High Risk