What is DQS and SSIS? First, to paraphrase from the Microsoft SQL Server Books Online … SQL Server Integration Services (SSIS) is a component of Microsoft SQL Server that is a platform for building enterprise-level data integration and data transformations solutions, used to solve complex business problems. SSIS can extract and transform data from a wide variety of sources such as databases, flat files, XML data files, Excel, and then load the data into one or more of these destinations. SSIS includes a rich set of built-in tasks and transformations and tools for constructing packages. You can use the graphical SSIS tools to create solutions without writing a single line of code, or you can code custom tasks and other package objects, and further you can develop programs to extend SSIS. To elaborate on the description on the Bentley web site ( http://www.bentley.com/en-US/Products/eB+Data+Quality+Manager/ ) ... Bentley Data Quality Server (DQS), released with eB Information Manager (eB) SELECTseries 4 (15.6.1.), uses and leverages SSIS and the functionality to "develop programs to extend SSIS" to provide a flexible and scalable data quality verification, bulk import and consolidation, and export functionality to a workgroup, project, or enterprise. DQS incorporates a proven engine that efficiently consolidates and integrates data from external sources to populate a central data store provided by eB through data extract > transform > load (ETL) processes. DQS SSIS System Input:: and Standard User:: Variables Namespace Name Data Type Description Source Input :: eBApplicationServer String eB Application Server name Execute SSIS Package Engine Input :: eBCommunityName String eB Community Name Execute SSIS Package Engine Input :: eBConnectInfo String eB Connection information, encrypted Execute SSIS Package Engine Input :: eBDataSource String eB Data Source name Execute SSIS Package Engine Input :: eBDBDatabase String eB Database name Execute SSIS Package Engine Input :: eBDBServer String eB Database Server name Execute SSIS Package Engine Input :: eBDBType String eB Database Type Execute SSIS Package Engine Input :: eBSessionID Int32 eB Session ID Execute SSIS Package Engine Input :: eBUserName String eB User Name Execute SSIS Package Engine Input :: WorkingDir String DQS Working Directory (with trailing "\" backslash) Execute SSIS Package Engine User :: WorkingDirInput String DQS Working Directory Input sub directory Expression† User :: WorkingDirOutput String DQS Working Directory Output sub directory Expression† User :: WorkingDirTemp String DQS Working Directory Temp sub directory Expression† User User :: :: ContextID DataStreamID String DQS Context ID date and time stamped† Expression*† Queue Job Attributes passed from the eB Execute SSIS Package Engine are not standard eB DQS system Input:: variables, User::DataStreamID is the recommended eB Context ID. SSIS Expressions Variable SSIS Expression Notes User::WorkingDirInput @[Input::WorkingDir] + "Input\\" Required sub directory name User::WorkingDirOutput @[Input::WorkingDir] + "Output\\" Required sub directory name User::WorkingDirTemp @[Input::WorkingDir] + "Temp\\" Required sub directory name User::ContextID User::DataStreamID Version 1 " " + "d" + RIGHT((DT_STR, 4, 1252)DATEPART("yyyy", @[System::StartTime]), 4) + RIGHT("0" + (DT_STR, 2, 1252)DATEPART("mm", @[System::StartTime]), 2) + RIGHT("0" + (DT_STR, 2, 1252)DATEPART("dd", @[System::StartTime]), 2) + "t" + RIGHT("0" + (DT_STR, 2, 1252)DATEPART("HH", @[System::StartTime]), 2) + RIGHT("0" + (DT_STR, 2, 1252)DATEPART("mi", @[System::StartTime]), 2) + RIGHT("0" + (DT_STR, 2, 1252)DATEPART("ss", @[System::StartTime]), 2) Version 2 @[System::PackageName] + " " + (DT_WSTR, 19)(DT_DBTIMESTAMP)@[System::StartTime] = short label, name, reference For the SSIS package When SSIS returns a failure status it intentionally fails the executed job. Below are two options on how to resolve the issue in PWECM to not cancel the whole package. Set the MaximumErrorCount to a higher number. This way the package will only fail if there are at least X number of errors reported Set the ForceExecutionResult to Success. This way the package will return a Success status no matter what. I prefer this since it is more predictable. Generate an error log in your SSIS package and put it in the Output folder. It will be added to the package document on completion. You can do this two ways. Use the built in SSIS logging capabilities. You can use XML or TXT and you can turn off most of the logging output so that it is not too cluttered. Below, I am only logging the SourceName and MessageText for the OnError event. Create an OnError event handler in SSIS and log it yourself. The default SSIS logging can be ugly even if you turn off most of the output. This solution allows you to create a cleaner file with only the information you want. You could even log the output to the status table or maybe even write directly to an attribute on the Package document. Andrew had developed an event hander that created a clean CSV file that could be opened in Excel. I hope to get that task and use it to create a custom task that we deliver with the software so you don’t have to do it yourself.
↧