BuER
Multilingual entity resolution pipeline linking 1.7M noisy business records
BuER is an entity resolution pipeline engineered for the Amazon ML Challenge 2026 to link noisy, multilingual business records across three disparate sources. It tackles combinatorial entity disambiguation through country-partitioned candidate retrieval, dictionary-backed Indic-to-Latin token transliteration, RapidFuzz string similarity, structural street-number conflict pruning, and an exclusive global assignment optimizer that assigns each candidate to its single highest-scoring entity.
Scale in entity resolution is not won by throwing monolithic transformers at pairwise comparisons; it is won by intelligent candidate blocking and deterministic conflict elimination before scoring begins.
Macro F0.5
0.987
author-reported competition score on unseen test split
Competition Rank
250
among 90,000+ participating engineers in Amazon ML Challenge
Submitted Pool
1.73M
source entities processed and disambiguated
The page keeps the strongest metrics and the strongest caveats in the same frame. Numbers are included because they clarify the system, not because they decorate it.
Match Links
5,803,442
exclusive assignments generated for final submission
Candidate Pairs
12.9M
7.46 candidates per source entity under posting frequency limits
Indic Lexicon
3,103 tokens
curated Indic transliteration dictionary with Unidecode fallback
The project is explained across seven recurring dimensions so the reader can understand not only what was built, but why the implementation matters, where the proof is strongest, and where the limits still are.
Case-study dimension
Why This Exists
Cross-source business entity resolution at national and global scale faces an exponential pairwise comparison problem: naive matching of 1.73 million entities against millions of candidate pool records requires trillions of comparisons.
BuER was engineered to solve this combinatorial explosion under competitive time and memory constraints, handling severely corrupted multilingual tokens, varying script encodings, and missing address components.
Case-study dimension
Why It Matters Now
Modern enterprise supply chains, tax fraud detection, and credit underwriting all depend on reliable company deduplication across noisy data streams without manual human curation.
As business data increasingly spans multilingual markets across Asia and the West, entity resolution systems must handle script divergence like Indic-to-Latin transliteration directly rather than assuming standardized English records.
Case-study dimension
What It Improves
The pipeline introduces country-partitioned blocking combined with token posting-frequency ceilings, pruning over 99.8% of the comparison space while retaining an average of 7.46 high-probability candidates per entity.
A custom 3,103-token Indic dictionary maps regional phonetic variations directly to normalized Latin roots, eliminating false negatives caused by script transliteration inconsistencies.
Case-study dimension
What Makes It Different
Rather than relying on greedy local thresholds, BuER enforces global exclusivity: each pool candidate record is assigned strictly to its single highest-scoring source entity.
Structural street-number compatibility checks run before fuzzy address distance calculations, guaranteeing that two companies sharing a street name but occupying different building numbers are never erroneously conflated.
Case-study dimension
Why It Is Hard To Replicate
The engineering difficulty lies in tuning blocking frequency thresholds to balance recall against memory footprint on 12.9 million candidate pairs.
Balancing weight between token-sort ratios, partial string matching, and transliteration confidence requires deep empirical optimization rather than generic heuristics.
Case-study dimension
How It Can Grow
The architecture is modular: the postprocessing optimizer already supports prefix selection, and could be extended with active-learning feedback loops or GPU-accelerated embedding retrieval for ambiguous edge cases.
Adding geographic coordinate geospatial bounding boxes would provide an additional physical validation layer for address matching.
Case-study dimension
Why It Lasts
Data noise in business directories is permanent: formatting changes, transcription typos, and translation divergences will always exist in real-world ingestion pipelines.
A deterministic, fast blocking and scoring architecture provides a durable, cost-effective baseline that runs in minutes on commodity hardware without cloud API dependencies.
Candidate Blocking
Country partitioning and posting limits reduce trillions of comparisons to 12.9M pairs
Pairwise entity comparison across 1.73M records is impossible without aggressive blocking. The indexer partitions records by ISO country code and indexes tokens in an inverted index with strict frequency thresholds.
Ubiquitous corporate suffixes like 'Pvt Ltd' or 'Enterprises' are pruned from the posting list, ensuring the candidate pool is populated only by discriminative business identifiers.
Multilingual Resolution
Deterministic transliteration bridges Indic script variations before fuzzy scoring
Indic company names frequently appear in varying scripts (Devanagari, Bengali, Telugu, Tamil) alongside English phonetic transcriptions. BuER uses a curated 3,103-token lexicon with Unidecode fallback to normalize phonetics.
RapidFuzz token-sort and partial ratios then compare normalized names and addresses, while structural regex guards flag building-number contradictions before score aggregation.
Next case
DALE