I encountered a weird problem with SSIS lookup transformation a few weeks back and today when working on a DW project. Lookup transformation, when run in full cache mode, caches the results during the pre-execution of the data flow into the memory. I prefer to use this method because Merge Join needs a sorted data set. The cost of sorting, whether we do it on SQL or SSIS, is expensive.
So, I was trying to join on smart key we were using on one of our legacy systems. This transformation did a looking for AccountID from the payment table, using the smart key (which is AlternateAccountID), and put the values in the payment table (child table). I found more than 1000 records for which I could not get any match. Before mapping this to the unknown dimension (-1), I decided to poke around and see why such a large result set would be present.
Ran a query against this orphaned data and the base table to see that many results are matching. It immediately hit my mind that SSIS lookup transformation, when run in Full Cache Mode is case sensitive, unlike SQL Statements. After looking at the speed of Partial Cache and No Cache Mode, I decided to change the smart keys to UPPER(smartkeys) in T-SQL. Presto! This seemed to bring down the problem to 200 orphaned records.
To sum it all up, SSIS Lookup Transformation when run in Full Cache Mode is case sensitive.
↧
SSIS Lookup Transformation
↧
