Principal Engineer AI

Pennymac (NYSE: PFSI) is a specialty financial services firm with a comprehensive mortgage platform and integrated business focused on the production and servicing of U.S. mortgage loans and the management of investments related to the U.S. mortgage market.

As the Principal Applied AI Engineer, you will be the lead architect of the "cognitive" layer of the AI platform. While the Backend Principal builds the deterministic infrastructure, you will engineer the probabilistic systems that power our "Unified Context Library" and "Agentic Orchestration" layer. You will design autonomous workflows that turn a Product Manager’s idea into a technical specification, and a User Story into deployable code.

• Architect Agentic Workflows: design and implement sophisticated multi-agent systems that can plan, execute, and self-correct complex tasks (e.g., automated code reviews, test plan generation, and epic decomposition).
• Develop orchestration flows using Lang Chain.ts and AWS Bedrock Agent Core, defining how agents hand off tasks to one another and when to involve humans for review.
• Engineer hallucination checkpoints and validation logic to ensure AI outputs are accurate, secure, and deterministic where necessary.
• Implement the Model Context Protocol (MCP) to standardize how agents interface with internal tools like Jira, Git Lab, and AWS infrastructure.
• Lead the strategy for our Retrieval-Augmented Generation (RAG) foundation, designing pipelines that ingest, chunk, and vectorize institutional knowledge from Confluence, Jira, and Git Lab.
• Optimize vector database performance (e.g., Pinecone, Weaviate) and implement advanced retrieval strategies (hybrid search, re‑ranking) to provide agents with precise, domain‑specific context.
• Build memory systems (short‑term and long‑term) that allow agents to retain context across long‑running sessions and provide personalized assistance to users.
• Design and maintain the Observability & Fine‑Tuning Framework, capturing every token, prompt, and user feedback signal to improve agent performance over time.
• Define and enforce prompt engineering best practices, creating a reusable library of system prompts that govern agent persona, tone, and output formatting.
• Build automated evaluation pipelines (using tools like Lang Smith or custom harnesses) to benchmark agent performance against golden datasets and prevent regression.
• Serve as the subject‑matter expert on generative AI for the Platform Services division, staying ahead of the curve on LLM capabilities, cost optimization, and model selection.
• Mentor fellow engineers on the paradigm shift from deterministic coding to probabilistic AI engineering.
• Drive the adoption of AI best practices across the wider organization.

• Bachelor’s Degree in Computer Science or equivalent, with 8+ years of professional software engineering experience.
• Expertise in Type Script/Node.js, with deep experience building backend services and AI chains.
• Hands‑on experience building applications powered by LLMs, including shipping products using Lang Chain, Strands, or AWS Bedrock.
• Proven track record of building production‑grade RAG systems; familiarity with embeddings, vector stores (Pinecone, Milvus), and semantic search.
• Extensive experience with AWS serverless architecture (Lambda, API Gateway, Dynamo
  
  DB); familiarity with AWS Bedrock and Agent Core is a significant advantage.
• Systems thinking: ability to design complex, asynchronous systems where state is fluid and outcomes are probabilistic.
• Startup mentality: high ownership, high energy, and the ability to thrive in a fast‑paced, internal startup environment.

• Experience with evaluation frameworks (e.g., Lang Smith, Ragas) for automated testing of LLM outputs.
• Familiarity with the Model Context Protocol (MCP) for standardizing AI tool connections.
• Background in developer tools (CLI tools, IDE plugins, CI/CD automations).
• Understanding of graph databases (e.g., Neo4j) for knowledge‑graph implementation alongside vector search.

As one of the top mortgage lenders in the country, Pennymac has helped over 4 million…