Background
I’ve been kicking around some ideas for what the Data Importer can become and wanted to put them here for some feedback/discussion.
Data importer can be an extremely valuable tool, especially since I believe we have to assume that Salesforce ISVs are not going to be building NPPatch specific integrations in the same way that some have NPSP specific integrations. I also think it can be so much more than just donations.
Proposal
I would love for us to reframe the data importer as the go-to place for ‘constituents plus other information’ imports AND integrations of all kinds. Nearly every nonprofit that we want to work with has spreadsheets that need importing into Salesforce that have firstname, lastname, email columns and then a bunch of other stuff (donations, event check ins, volunteer sign ups, etc).
Additionally, a lot of third-party platforms offer Salesforce ‘integrations’ that are really just Zapier or something similar. I would love our users to use a single zap (or equivalent) to put that data on a data import object and then be able to use point and click mapping, rather than trying to figure out how to do custom matching, mapping, and data transformation elsewhere.
Use Cases
A couple use cases that I’ve done myself using the NPSP Data Import functionality below for illustration.
-
An org with a large direct mail program imported the donation file from their mailhouse with zero initial excel work. Flows on the Data Importer record split the full donation coding into constituent + campaign + appeal + package codes for matching against existing Salesforce records. Coding mismatches for campaigns, appeals, and packages were flagged on the Data Import record for manual review before the batch was processed.
-
Event registrations from a third-party app created ‘Registration’ records in Salesforce. A triggered flow copied the information to one or more Data Import records on a daily import batch depending on whether an additional donation was given and/or the registrant was the same person as the payee. Constituent Ids from the third party system were used to match existing Salesforce contacts and prevent duplicates, while creating the correct opportunities and campaign membership records to track event participation. All of the advanced logic (eg. whether to create one or two records) is an a single flow, and the field mapping is in a single transform element. Again, the ‘raw’ information is available on the Data Import object, and the admin was able to spend some time manually reviewing and processing each batch manually to build confidence before the daily batches were set to process overnight automatically.
Technical Blockers/Issues
The Data Importer is close to good enough to do all of the use cases above (and more) without workarounds, but there are some issues.
- Currently, the importer will ALWAYS attempt to import any records that are not already marked as ‘Imported’. That means that you can flag a records as ‘Failed’ because of some custom matching or validation failure, but the data importer logic will attempt an import anyways. For example, in the direct mail use case above, code mismatches were flagged, but there was no technical way to prevent donations from being created anyways (unless the constituent mismatch meant that NPSP was trying to create a donation with no donor), so there was always a risk that a missed coding error flagged before the import could result in an opportunity not being connected to the correct campaign or appeal.
- There are also some restrictions built into the matching logic for donations because of the assumption that it is only ever used in a Gift Entry scenario. Specifically, there is a hard-coded query that will only every match open opportunities. So an integration that is trying to process a refund on an existing opportunity cannot use the built in functionality.
- There is no easy way to build in additional matching logic, which is especially tricky for something like Campaigns. Right now the importer matches campaigns on name only, which is problematic.
- Because all of the workarounds for matching are flow or apex based (and usually record-triggered), errors can block the creation of data import records entirely, which makes their value as an audit trail more difficult. It can also have processing/performance implications.
There is some extensibility in the Data Importer functionality currently with the ability to add a custom Donation Matching Implementing Class or a Post Processing Class, but it’s not quite enough.
Technical Proposal
We should create a framework/interface similar to TDTM for pre/post processing apex classes or flows inspired in part by TDTM, and move the existing contact and donation matching into that framework.
What this looks like for end-users
- Advanced Mapping functionality stays the same
- An additional settings page is added for managing Batch Processing Logic entries (similar to TDTM). Each entry has the Apex class or Flow Name, an execution order, and whether the class/flow is for pre or post processing
- Out of the box, NPPatch has two Batch Processing Logic entries: one for contact matching and one for donation matching.
What this looks like technically
The batch processing logic interface requires a single Processing() method with the dryRun boolean and a list of data import records, similar to the existing BDI_DataImportService and BDI_IPostProcess classes. Being able to use a flow instead of Apex would be very cool, but is a lower priority in my opinion.
During the import process, the logic:
- Cycles through all pre-processing classes and flows
- Handles the Advanced Mapping logic for creating/updating existing records based on matching
- Cycles through all post-processing classes
What this enables us to add/update more easily in the future
- External Id matching for Recurring Donation and Campaign objects
- A toggle to enable/disable matching on closed/won opportunities
- Support for PMM objects