Own the AI safety layer for Realtor.com’s RealAssist, an LLM-powered real-estate assistant used across web and mobile. On the AI Integrations Team at News Corp, you will build and tune LLM safety guardrails that act as a launch gate for Fair Housing compliance and content moderation, with runtime prompt-injection and jailbreak screening designed for production.
What you’ll do
- Build, tune, and take end-to-end ownership of LLM-as-a-judge classifiers for Fair-Housing compliance and content moderation, targeting high recall on disallowed content without overblocking legitimate users
- Design and run guardrail evaluation pipelines, including labeled dataset curation, train/test/validate splits, and offline prod-replay to evaluate overblocking
- Use defensible metrics by reporting confusion matrices and precision/recall/F1 per category to guide safety decisions
- Integrate and operate cloud guardrail services as a runtime prompt-injection/jailbreak screening layer, including fail-open behavior and alerting
- Red-team the assistant and wire safety regression checks into CI to prevent silent guardrail degradation as the product grows
- Manage guardrail infrastructure as code using Terraform (templates, IAM, and project shape), and participate in technical design reviews and architecture discussions
- Partner with ML, backend, product, and legal/compliance stakeholders to define risk tiering and get new AI capabilities through safety review before launch
- Independently design, implement, and tune guardrail and evaluation systems with a focus on balancing safety, latency, and user experience; write clean, maintainable, well-tested code
- Keep eval datasets calibrated to real traffic and emerging attack patterns, staying current with AI-safety research and new guardrail tooling
What you bring
- AI Safety & Evaluation Expertise (required)
- 4+ years of professional software/ML engineering experience with hands-on LLM application work
- Bachelor’s degree or equivalent experience
- Strong Python proficiency
- Real classification-metrics literacy: precision/recall/F1 (macro vs weighted), confusion matrix debugging, dataset curation, and calibration to production distribution
- Experience evaluating LLM prompts/classifiers with an eval framework such as DeepEval, RAGAS, Arize Phoenix, LangSmith, OpenAI Evals, or a homegrown harness
- Ability to turn small seed sets into robust labeled eval datasets using dataset-synthesis and prompt-optimization loops
- Familiarity with prompt-injection/jailbreak defense concepts, including OWASP LLM Top 10, input/output filtering, least-privilege tool access, and adversarial testing
- Hands-on experience with cloud guardrail services: Google Cloud Model Armor, AWS Bedrock Guardrails, or Azure AI Content Safety
- Terraform/IaC for cloud infrastructure
- Experience with Google Cloud Vertex AI / Gemini
- Understanding of monitoring and observability tools (for example, New Relic or similar)
- Exposure to regulated, compliance-sensitive domains (fair housing, fair lending, healthcare, finance, trust & safety)
- Red-teaming or AI-security background
Tools you’ll work with
Python, DeepEval, RAGAS, Arize Phoenix, LangSmith, OpenAI Evals, OWASP LLM Top 10, Google Cloud Model Armor, AWS Bedrock Guardrails, Azure AI Content Safety, Terraform, Google Cloud Vertex AI / Gemini, New Relic
Onsite role with strong support and collaboration
The AI Integrations Team is a cross-functional squad building how Realtor.com integrates AI across its products. The work blends creativity and innovation with in-person collaboration. For most roles, employees work four or more days in office.
Benefits
- Inclusive and competitive medical, Rx, dental, and vision coverage
- Family forming benefits
- 13 Paid Holidays
- Flexible Time Off
- 8 hours of paid volunteer time off
- Immediate eligibility into the Company 401(k) plan with 3.5% company match
- Tuition Reimbursement for degreed and non-degreed programs
- 1:1 personalized Financial Planning Sessions
- Student Debt Retirement Savings Match program
- Free snacks and refreshments in each office location
