AI Quality Testing Framework
The AIQ Testing Framework operationalises established standards into a streamlined audit procedure that makes the quality, reliability and organisational maturity of AI systems transparent.
Focus
- Auditable quality characteristics
- Structured evidence provision
- Management-ready results
Audit process:
Five steps towards a robust assessment of AI systems.
The framework combines governance, technical evidence and transparent and understandable evaluation in a compact process that can be used operationally.
1
Use-case analysis
Clear classification of the AI system, its context of use and its intended purpose. Capture of relevant risks relating to quality, safety, governance and trustworthy use.
2
Definition of quality requirements
Derivation of specific and auditable requirements as the basis for the subsequent assessment.
3
Provision of evidence
Collection of documentation, test reports and further robust evidence for the AI system.
4
Validation of evidence
Assessment of the completeness, robustness and traceability of the evidence provided.
5
Preparation of the AIQ audit report
Consolidation of the results in a structured report with assessment, classification and recommendations.
Quality dimensions:
What the assessment looks at in concrete terms.
The assessment focuses on six quality dimensions and maps each of them to specific requirements of the EU AI Act.
Reliability
EU AI Act: Art. 15 | Art. 17 | Art. 72 | Art. 73
The assessment examines whether the AI system functions robustly, reproducibly and reliably in practical use.
Data quality, data protection and data governance
EU AI Act: Art. 10 | Art. 13
The focus is on the quality, origin, governance and protection of data throughout the AI system lifecycle.
Transparency
EU AI Act: Art. 11 | Art. 12 | Art. 13 | Art. 50
It is assessed whether functionality, limitations, documentation and information obligations are implemented clearly and appropriately.
AI-specific cyber security
EU AI Act: Art. 15 | Art. 53/55
Technical and organisational measures to protect against manipulation, misuse and security-relevant vulnerabilities are considered.
Non-discrimination
EU AI Act: Art. 10 | Art. 15 | Art. 53/55
The assessment reviews whether risks of unfair bias are recognised, reduced and addressed transparently through suitable controls.
Human oversight and control
EU AI Act: Art. 14 | Art. 53/55 | Art. 72
It is evaluated how effectively humans remain involved in critical decisions and how interventions and escalations are organised.
Assessment scenarios:
Use cases
The following examples show practical application scenarios in which companies can particularly benefit from a structured quality assessment of their AI systems.
Healthcare / Medtech
Medical – Radiological image analysis
An AI system analyses X-ray or CT images and supports radiologists in identifying potential abnormalities.
Computer Vision
High risk (EU AIA)
How our framework helps:
- Model validation: Comprehensive assessment of sensitivity, specificity and robustness across different image types and patient populations.
- Bias analysis: Review of discrimination against patient groups, age groups and sex.
- EU AI Act compliance: Documentation to demonstrate fulfilment of highly regulated requirements and risk classification.
- Explainability: Validation of the traceability of model predictions for clinical use.
- Monitoring: Establishment of KPIs and control mechanisms for continuous quality monitoring.
Finance / Banking
Finance – Credit risk assessment
A financial institution uses machine learning to assess credit risks and to automate the preliminary decision on credit applications.
Classical ML models
High risk (EU AIA)
How our framework helps:
- Fairness testing: Comprehensive review of discrimination based on protected characteristics (sex, age, ethnicity).
- Model robustness: Validation of model stability across different economic cycles and customer groups.
- Regulatory compliance: Evidence for supervisory authorities (ECB, BaFin) to demonstrate fulfilment of risk management requirements.
- Explainability: Provision of comprehensible reasons for credit decisions to meet transparency requirements.
- Stability testing: Testing for overfitting and robustness in adversarial scenarios.
SaaS / Digital Services
Customer support – LLM chatbot
A company uses an LLM-based chat system to answer customer enquiries automatically.
How our framework helps:
- Hallucination testing: Systematic review for fabricated or incorrect information in responses.
- Bias and fairness review: Assessment for stereotypical or unfair responses towards particular user groups.
- Data security: Validation that sensitive customer data are not disclosed through training data or outputs.
- Robustness: Testing of edge cases and adversarial prompts to ensure system security.
- Performance and quality: Definition of metrics for response quality, customer understanding and escalation rates.
LLM
Limited risk (EU AIA)
Insurance
Insurance – Document analysis
An AI system analyses claims notifications and insurance documents and supports case handlers in processing them.
LLM /
Document AI
Limited risk (EU AIA)
How our framework helps:
- Data quality assessment: Validation of training data for completeness, consistency and representative coverage of different claim types.
- Accuracy assessment: Review of extraction accuracy for critical information (claims, amounts, policies).
- Fairness assessment: Verification that the system does not systematically disadvantage particular customer groups.
- Data protection compliance: Evidence that personal data are protected appropriately.
- Governance: Documentation of responsibilities and escalation processes for error handling.
E-Commerce
Retail – Product recommendation system
An online shop uses AI for personalised product recommendations and to optimise the customer journey.
Recommen-dation system
Low risk (EU AIA)
How our framework helps:
- Fairness of recommendations: Assessment that the system does not systematically favour or disadvantage particular product categories or suppliers.
- Data quality: Review of training and reference data for bias and representativeness.
- Data protection compliance: Evidence of appropriate handling of customer data and tracking compliance (GDPR).
- Transparency: Documentation of how recommendations are generated and which factors influence them.
- Performance monitoring: Definition and monitoring of KPIs (CTR, conversion, customer satisfaction) for continuous quality improvement.
HR / Recruiting
HR – Applicant screening
A company uses AI for the preliminary analysis of applications and to support candidate selection.
How our framework helps:
- Discrimination testing: Comprehensive analysis for bias relating to sex, age, ethnic origin, disability and other protected characteristics.
- Fairness assessment: Validation that the system offers equal opportunities to qualified candidates from different backgrounds.
- Transparency and traceability: Documentation of how the assessment is carried out and which criteria lead to rejection.
- Legal compliance: Evidence to demonstrate fulfilment of EU AI Act requirements.
- Monitoring and governance: Establishment of review processes and KPIs for continuous fairness monitoring.
NLP / ML
Limited risk (EU AIA)
Aligned with established standards
The framework does not stand in isolation; instead, it aligns evidence with established regulatory and normative frameworks.
Testing
Framework
Regulatory framework for requirements for trustworthy and secure AI systems.
Risk-oriented approach for governance, measurability and ongoing control of AI.
Management system for responsible development and use of AI.
Structured guideline for AI risk management.
International principles for transparency, fairness, robustness and accountability.
Reference framework for structured evaluation and further development of quality in AI systems.
Choose the right audit for your AI system
The two audit options focus on organisational quality and, additionally, on technical evidence.
Option 1
AIQ Audit
- Quality management
- Governance and processes
- Documentation and monitoring
- Structured evidence review.
To make organisational maturity and quality status transparent.
Option 2
AIQ Technical Audit
- Includes all elements of the AIQ Audit, plus evaluation of technical evidence
- Performance metrics review
- Robustness testing
- Bias analyses.
To verify technical quality and system performance independently.
So können sie uns erreichen:
Our Shareholders
Get to Know Us
We look forward to engaging with you. Let us work together to bring AI ideas and projects to life. Contact us to arrange a non-binding initial consultation!