Avoid processing of unchanged records

DataBridge subscriptions that are entered after October 1, 2021 come with 50.000 records per month to process. For all processed records above that limit, additional costs will incur.

Before we start looking into optimizing the ‘flow’ of records, we’ll start with the way processed records are counted.

Any record that is ‘flowing through’ DataBridge is counted as one record. Such a record can contain only company details or sale details, but it can also contain details of both. Or even more.

Let’s try to explain this with an example. You are aiming for an import of companies and contacts and from Excel to SuperOffice. Every row contains details from both the company and the contact. With the details in such a single row, DataBridge can create or update two entities: a company and a contact. If you combine records, you will save the registered number of processed records.

Numbers to illustrate

Let’s compare the numbers here. For the sake of keeping the numbers simple, we’ll assume that we have 1.000 contacts at 500 different companies.
If we have all details in one file, the file would consist of 1.000 records. Every time DataBridge runs the profile that would be 1.000 processed records added to your record counter.
How does that look once we have the files split up and we have one source file with 500 companies and one source file with 1.000 contacts? That would mean that we have two profiles: one for the company import and one for the contact import. Every time those profiles run there are 1.500 records added to your record counter.

Let’s dive a bit deeper into this example and see how the number of runs influences our monthly record counter. The main question here usually is: how up-to-date do you need to keep the data in SuperOffice CRM? The result is the number of processed records for a month of 30 days.

In the example, we don’t care about whether the details of a company or person are updated or new since the last import. We just export them all from our source without any filter so every run results in the same number of rows. Over time the number of rows will most likely grow since we will add new companies and contacts.

Changed records only

To continue our challenge for optimization of our import, we requested the team that takes care of the export of the source files to only deliver the changed or newly created records. They already responded that this is possible and that such a file will be delivered automatically every hour.

What would this change in file generation mean for the processed record counter? To see the results in numbers, we need to assume how many of our records would change every month. Let’s set the percentage of changed records each month to a whopping 50%.

Now back to the numbers from our example: 1000 contacts at 500 companies. If 50% changes in a month, then we would process 500 contacts records and 250 company records every month. If these records are all in one file, then we would have the same number of changes, since probably the company details change at a different date and time than the contact details.

The difference is very clear. To avoid additional costs it is a very good idea to get your source system to produce exports with the changed records only. In that case, it probably does not really matter if you have several entities in one file or split up over several files.

Please note that the initial import would need to import all records one time. If the number of records that you need to process is much more than the numbers in the example, your first bill might be a bit higher.

Export from SuperOffice

The above example focuses on importing data into SuperOffice. We also see a lot of profiles that export data from SuperOffice. For instance to populate an ERP platform or a BI data source. It seems a very simple solution to export all records once every day. However, if we look at the numbers from our example, the 30-day results would match the numbers that we see for importing into SuperOffice.

All exports from SuperOffice are based on a SuperOffice Selection. DataBridge has a feature to only retrieve the data (records) that was updated or new since the last successful DataBridge job. It’s good practice to use this feature to avoid having too many processed records added to your monthly counter. You’ll find this feature in the ‘Configure Source’ tab in the ‘Advanced Settings’ section.

Open

Enabling this feature will make sure your processed records counter will not be higher than necessary.