top of page
1 Carlos Ayala Logo Oficial Grande Con Contorno .png
OPPORTUNITIES ARE HERE AND THEY ARE MEANT TO BE TAKEN 
WOULD YOU?
Before the interview
AI QA Engineer (Accuracy & Reliability)

Full Time

Business Development Quality Assurance, Engineering

Mexico

Salary Range: US$7,500.00 to US$10,000.00* per Monthly

*Depending on experience, capabilities & performance

CA-IT-IA-1122

Job Description

This is a pioneering role at the intersection of traditional Quality Assurance and modern Data Science. In the context of a national Real Estate network, a "small error" in an AI's rent projection or a misunderstanding of a lending term can have significant legal and financial consequences. You will design and execute complex testing frameworks specifically built for Anthropic’s Claude models. You will spend your days simulating edge-case conversations, evaluating the accuracy of Retrieval-Augmented Generation (RAG) systems (ensuring the AI only uses the data we provide), and building automated "red-teaming" scripts to try and trick the AI into giving incorrect or biased information. You will collaborate closely with the Prompt Engineering and Data teams to identify where the model fails and provide data-driven feedback to improve the system's reliability across thousands of property listings and diverse service categories.

Key Responsibilities:

Automated API testing, data truth verification (Grounding), and regression testing.


  • Accuracy Benchmarking: Establish and maintain "Golden Datasets"—sets of perfect answers that the AI must match to ensure it isn't drifting in quality.

  • Hallucination Monitoring: Implement systems to detect when the AI invents property details or financial terms that are not present in the source databases.

  • System Integration Testing: Verify that the "hand-off" between Claude and our internal APIs (MLS feeds, Lender databases) is seamless and data remains uncorrupted.

  • Bias & Safety Auditing: Regularly audit the AI’s responses to ensure it adheres to "Constitutional AI" principles and Fair Housing laws, preventing any discriminatory output.

  • Reporting: Translate complex AI performance metrics into clear, actionable reports for the Product and Engineering teams.

Key Activities:

Creating test scripts, reporting bugs in Claude integrations.

  • Automated Testing Suite Development: Writing Python scripts (using PyTest or similar) to run hundreds of simultaneous queries against the Claude API.

  • RAG Evaluation: Using tools like RAGAS or TruLens to measure "faithfulness" (is the answer based on the doc?) and "relevancy" (does it answer the user's question?).

  • Manual Red-Teaming: Engaging in "adversarial" chats with the AI to find vulnerabilities in its logic or safety guardrails.

  • Bug Life-Cycle Management: Identifying, documenting, and tracking AI-specific bugs (e.g., "model is too verbose," "incorrect calculation of IRR," "listing data mismatch").

  • Collaboration: Attending daily stand-ups to provide the "quality perspective" on new feature deployments for the Property Management and Syndication modules.

Academic Skills and Qualifications

  • Bachelor’s Degree: Computer Science, Information Technology, Mathematics, or a related technical field is required.

  • AI Specialization: Certifications or specialized coursework in Machine Learning, Natural Language Processing (NLP), or LLM Operations (LLMOps).

  • Logic & Linguistics: A strong academic foundation in formal logic or computational linguistics is a major plus, as it helps in understanding how Claude processes semantic meaning.

  • Real Estate Familiarity: While not mandatory, a basic understanding of real estate terminology (escrow, cap rates, syndication) is highly valued to better design test cases.

Key Skills and Qualifications:

Selenium, PyTest, LLM Evaluation Frameworks (e.g., RAGAS).


  • Technical Proficiency: Advanced Python skills and experience with SQL for data verification.

  • AI Tooling: Experience with LLM evaluation frameworks (e.g., DeepEval, Giskard, or Promptfoo).

  • API Mastery: Deep understanding of RESTful APIs and how to test them (Postman, Insomnia).

  • Analytical Mindset: Exceptional attention to detail; the ability to spot a single incorrect digit in a complex financial projection.

  • Communication: The ability to explain to non-technical stakeholders why an AI model might be behaving unexpectedly and the steps needed to fix it.

  • Adaptability: Since AI tech moves fast, you must be a "lifelong learner" capable of mastering new Anthropic model updates (like new versions of Claude) the day they are released.

Work Experience:

3+ years in QA Engineering.

Daily Job Schedule:

The daily job schedule will follow the standard working hours of the company, typically from 9:00 AM to 5:00 PM. However, occasional flexibility may be required to meet project deadlines or collaborate with remote team members.

What we offer!

At our organization, we work hard to establish a welcoming environment that draws in and keeps great people. We are aware of the importance of a supportive workplace environment for our employees' success and happiness. Here is a summary of the ideal workplace culture we provide to attract all job hopefuls:
 

  1. An inclusive and diverse culture is important to us because we think that different viewpoints and experiences foster better creativity and problem-solving. Every employee feels accepted, valued, and empowered to offer their special thoughts and experiences because we promote an inclusive atmosphere.

  2. We encourage teamwork and collaboration because we know that when people work together, they typically get the best results. To create a friendly and supportive workplace, we promote open communication, idea exchange, and cross-functional cooperation.

  3. Opportunities for Growth and Development: We place a high priority on our workers' continued development. To improve their skills and expertise, we provide a range of learning and development programs, mentorship opportunities, and ongoing training. We give employees a clear career path and aid them in reaching their professional objectives.

  4. Work-Life Balance: We are aware of how crucial it is to keep a positive work-life balance. We encourage employees to take time off to recover and pursue personal interests and we provide flexible work schedules, remote work opportunities (where appropriate), and other benefits. We think that having a healthy work-life balance makes people happier and more effective.

  5. Recognizing and rewarding hard work and contributions from employees is important to us. We offer benefits and attractive salary packages that are in line with industry standards, and we have a structured recognition program that recognizes excellent achievement.

  6. Employee Support: We place a high priority on our employees' support and well-being. We provide access to services for both physical and mental health as well as comprehensive health insurance policies, wellness initiatives, and programs. We encourage staff to express their concerns or ask for help when necessary and maintain an open-door policy.

  7. Environment for Innovation and Creativity: We promote an atmosphere for innovation and creativity. We give workers the freedom, autonomy, and resources they need to try out novel concepts, innovative ideas, and cutting-edge technologies. We support questioning the status quo and making continual improvements to our procedures and products.

  8. Social responsibility and sustainability are important to us because we want to have a beneficial influence on both people and the environment. We offer volunteer opportunities, sustainability programs, and corporate social responsibility activities so that staff members can support worthwhile causes and change the world.

  9. Transparent Communication: We support open lines of communication and educating staff on new initiatives, objectives, and tactics. We have regular town halls, team meetings, and offer avenues for comments and feedback. We value candid conversation and welcome employee ideas and input.

  10. Fun and Engaging Activities: We are committed to fostering a positive work environment. We plan social events, team-building exercises, and joint milestone and achievement celebrations. We support an upbeat, welcoming environment where humor and creativity flourish.

Benefits

Health & Wellness

Fitness & Recreation

insurance.png

Health Insurance

healthcare.png

Life Insurance

Continuing Education

online-learning.png

Training & General Education

education.png

Foreign Language

Family

flexible.png

Flexible Scheduling

baby.png

Child Care

Finance

retirement.png

Retirement Plan

gift-card.png

Bonus / Compesation

Free Time

day-off.png

Paid Off Time

insurance.png

Sick Leave

Law Enforcement & Veteran Benefits

venta.png

Special Discounts

ensenanzas.png

Special Learning Programs

DISCLAIMER

We are proud to foster a workplace free from discrimination. We strongly believe that diversity of experience, perspectives, and background will lead to a better environment for our employees and a better product for our users and our creators. This is something we value deeply and we encourage everyone to come be a part of changing the way the world.

IMPORTANT: earnings and legal disclaimers Carlos Ayala is an Internet marketing and security professional and his results are not typical. Your experiences are not a guarantee that you will earn money. You can do more, less or the same or nothing at all. This is purely educational. No income is guaranteed.

Only serious and ambitious entrepreneurs please apply.

Prices are subject to change without notice. No refunds will be allowed for tickets at any price level. If for some reason you cannot attend the events that are organized, you will receive full credit for your investment in my store (iamcarlosayala.com). There is no risk for your investment with me today.

© Copyright 2025. All rights reserved | Much of the content on this website belongs to CAS and is protected by copyright laws. Content that is not owned by CAS is stated in their respective legal instruments.
© Copyright 2018 - 2024. All rights reserved | Much of the content on this website belongs to CAS and is protected by copyright laws. Content that is not owned by CAS is stated in their respective legal instruments.

© 2026 CAS Training Technologies.

bottom of page