DSS Matching Logic

## **Analysis and Improvement of DSS Invoice Matching Logic** The meeting centered on a deep technical dive into the Data Services System (DSS) invoice matching logic, with the primary objective of diagnosing persistent issues causing incorrect line item and meter creation. The discussion deconstructed the current multi-phase workflow to identify root causes in vendor matching, data normalization, and the overall integration with the AI (LLM) extraction process, concluding that a fundamental paradigm shift is required to improve accuracy. ### **Overall Problem Statement and Initial Focus** The core issue is that the system frequently fails to correctly match incoming invoice data with existing database templates, leading to the erroneous creation of new meters and line items instead of mapping to known ones. This results in data duplication and workflow inefficiencies. The immediate diagnostic focus was placed on three key areas: improving the core matching logic with potential "fuzzy" matching, cleaning up legacy data, and defining a clear process for handling invoices without an exact template match. ### **Deconstructing the Existing Matching Workflow (Phase 1)** The team meticulously walked through the documented matching flowchart to understand each failure point. - **Step 1: Vendor ID and Account Number Matching:** This initial gate relies on a vendor ID and account number. The analysis revealed critical vulnerabilities: these identifiers can be captured at upload or inferred post-IPS extraction, leading to potential mismatches. A major concern is the translation of a vendor *name* (from IPS) to a vendor *ID* (a number used for database lookup), which may fail if not an exact match. Furthermore, account numbers could be captured differently (e.g., with dashes or leading zeros), requiring normalization to ensure reliable string matching. If this step fails, the process cannot proceed to use an existing template. - **Step 2 & 3: Service Account and Meter Fetching:** Upon successful vendor/account matching, the system fetches all related service accounts (e.g., "Full Service," "Distribution Only") and their associated meters. The logic assumes that for existing customers, these entries should always be present. The discussion confirmed that the system pulls *all* service accounts and meters for the matched account, without initial filtering, which is a potential point for future optimization. - **Step 4: Line Item Fetching and Failure Paths:** Line items are fetched specific to each meter. The meeting clarified the hierarchical dependency: if vendor/account matching fails, meter matching inherently fails, leading to the creation of new structures from IPS data. This results in a bill being created without a linked customer account (`client_id` set to `null`), sending it to a manual review queue. ### **Identifying Specific Matching Failures** The conversation pinpointed exact scenarios where the current logic breaks down, using a real invoice example. - **Meter-Level Matching Criteria:** Meters are matched using a combination of **commodity description**, **rate code description**, and **service address**. The example showed that while the commodity matched, the rate code failed because the IPS extraction captured "Commercial Service" while the database record contained the full string "Rate 4: Commercial Service." This exact string mismatch caused a new meter to be created. - **Issues with Extracted Data:** The **rate code** and **service address** are particularly prone to extraction variance by the LLM. The address, especially, requires "fuzzy" logic for matching, as minor differences in formatting can cause failures. The **commodity description**, while more stable, can also be misidentified. ### **Core Flaw: Disconnection Between Template and AI** A critical insight emerged regarding the fundamental design of the comparison phase (Phase 2). - **Phase 2 (Comparison) Limitations:** The system compares the IPS-extracted "enhanced JSON" with the database template. However, it was determined that the process only *overwrites* the IPS data with database values **when an exact match is already found**. The database template does not actively *guide* or *constrain* the initial LLM extraction. - **Circular Logic Problem:** Essentially, the system uses the database to clean up data only after the AI has already made its best guess. If the AI's extraction is poor (e.g., on rate code), no match occurs, and the incorrect AI data is kept, leading to new entries. The existing template is not being utilized as a prior input to make the AI's extraction more accurate in the first place. ### **Proposed Solutions and Strategic Rethinking** The conclusion was that incremental tweaks are insufficient; a new approach is needed to tightly couple the known template data with the AI processing stage. - **Feed Templates to the LLM:** The proposed solution is to dynamically feed the known invoice template for a matched vendor/account **into the LLM system prompt** *before* extraction. This would instruct the AI to use the existing line item descriptions, commodity types, and rate codes as a reference, forcing it to align its output with the known structure. - **Implement Confidence Scoring:** The enhanced system should include confidence scores from the LLM. For example, extractions with less than 90% confidence could be flagged for review, and those below a lower threshold could require mandatory manual intervention, preventing incorrect automated creation. - **Ownership and Research:** The team resolved to take full ownership of this logic. Immediate next steps involve a thorough review of the existing LLM system and user prompts, the template matching code, and the exact method by which vendor names are translated to IDs. ### **Action Plan and Administrative Notes** The session concluded with a clear action plan and ancillary discussions about tooling access. - **Immediate Technical Investigations:** The team will review three key areas: the LLM system/user prompts governing extraction, the logic for translating vendor names to IDs, and the overall design of the template matching and comparison workflow. A meeting with the original developer was also scheduled to gain historical context. - **Development and Testing Path:** Changes to the LLM prompts can be made by cloning the repository, creating a feature branch, and deploying to a development environment for testing. This allows for controlled experimentation with new prompting strategies. - **Access and Infrastructure Issues:** A significant blocking issue is the inability to access internal tools (like FTG Connect) due to VPN/client conflicts on non-company hardware. This hinders the ability to test changes and investigate issues directly, and resolving this access problem was noted as a high priority for enabling effective development.