Sync Filter and incremental sync

Combining Sync Filter and incremental synchronization is not without problems.

Analysis of the sync behavior

The basic idea behind incremental sync is simple - we ask for all records changed since the last sync. So we usually see fetches such as:

<fetch version="1.0">
	<entity name="lead">
		<filter>
			<condition attribute="versionnumber" operator="ge" value="13486544" />
		</filter>
	</entity>
</fetch>

(We used the versionnumber from the last successful sync for this entity.)

Alternatively, we can see time-based fetches. This happens when the entity does not have the versionnumber field. (There are a few such entities.)

<fetch version="1.0">
	<entity name="entity">
		<filter>
			<condition attribute="modifiedon" operator="ge" value="2017-09-26T15:34:27.129Z" />
		</filter>
	</entity>
</fetch>

Notice that the above fetches do not have any relation to the entity sync filter. The reason is simple: If we used sync filter then we would never be notified that the server modified a particular record in such a way that it stopped matching sync filter.

For example, suppose our sync filter excluded inactive accounts:

<fetch version="1.0">
	<entity name="account">
		<filter type="and">
			<condition attribute="statuscode" operator="ne" value="2"/>
		</filter>
	</entity>
</fetch>

During the last sync, we downloaded all such record(s). But afterward, the server deactivated some of them. If we used a fetch based on the sync filter (shown in the following example) during the next incremental sync, the client would miss the information about deactivated accounts.

Therefore, the general philosophy adopted in the incremental sync is:

1. Get all changed records.
2. Drop records that do not match sync filter. (Done in sync cleanup.)

However, this procedure still has deficiencies:

• It misses record deletions. Same for ownership changes, changes in record sharing, permission changes, etc. This problem goes beyond the scope of this article and is discussed elsewhere. (Delete plugin, SyncShared; in the case of permissions changes, you are out of luck.)
• A record might get into view (start matching the sync filter) even if it was not modified.

Example: A salesorder might become activated and need to be downloaded to the client. However, its salesorderdetail records were unchanged.

• If there are too many changes, the algorithm may prove to be too inefficient.

In some scenarios, the client downloads millions of records only to delete 99% of them in the sync cleanup.

Resco offers 2 options that are designed to cope with the above problems:

• Incremental sync filters
• Linked sync filters

They solve different problems and you cannot use both of them: If you do so, then only the first option (incremental sync filters) will be realized.

Using Sync Filter in incremental sync

This option can be enabled in entity settings.
Internal name: EntityAttributes.IncrementalSyncFilter

For each entity, you can configure to use Sync Filter also during incremental synchronization: set Synchronization to "Incremental with Sync Filter". As you'll see later, a more precise description would be "Use FullSync fetch for incremental sync".

The reason to use this option is to reduce unneeded downloads, i.e., the third problem from the list above. Don't use it if you don't see this effect.

For example, if we want only active accounts, the IncSync may look like

<fetch version="1.0">
	<entity name="account">
		<filter type="and">
			<condition attribute="statecode" operator="eq" value="0"/>
			<condition attribute="versionnumber" operator="ge" value="13486549" />
		</filter>
	</entity>
</fetch>

As you can see, it is FullSync fetch + versionnumber condition.

Let's say right now that the above fetch represents a pure example for this option. If we don't have a guarantee that another client (or server) won't deactivate our account records, then we should not use it.

On the other hand, if the sync filter was limited to "my accounts" only, then the option would work better:

<fetch version="1.0">
	<entity name="account">
		<filter type="and">
			<condition attribute="ownerid" operator="eq-userid" />
			<condition attribute="versionnumber" operator="ge" value="13486549" />
		</filter>
	</entity>
</fetch>

But again: If the account ownership was taken away from the mobile user, the above fetch would miss this information.

In general, the option can be considered safe if we have reasonable guarantee that no other user will modify server records in a way that they stop matching our sync filter.

What happens to child entities? Normal IncSync fetch is constructed as always; for example:

<fetch version="1.0">
	<entity name="quotedetail">
		<filter type="and">
			<condition attribute="versionnumber" operator="ge" value="13486549" />
		</filter>
	</entity>
</fetch>

If the quotedetail entity has the IncrementalSyncFilter attribute, the SyncEngine simply adds an inner link to the parent with its sync filter, for example:

<fetch version="1.0">
	<entity name="quotedetail">
		<filter type="and">
			<condition attribute="versionnumber" operator="ge" value="13486549" />
		</filter>
		<link-entity name="quote" from="quoteid" to="quoteid">
			<filter type="and">
				<condition attribute="modifiedon" operator="last-x-months" value="24"/>
			</filter>
		</link-entity>
	</entity>
</fetch>

If an entity has immutable records (records that are never edited), then IncrementalSyncFilter will work without problems. The same is true if the entity records are "private" in a sense that they are edited by a single user only. In all other cases you have to weigh potential gains vs risks.

Linked Sync Filters

This option can be enabled in entity settings.

For each entity, you can configure to use linked Sync Filters: set Incremental Linked SyncFilter to "Enable".

Use case 1: Get active contacts belonging to active accounts

Here are the sync filters:

<fetch version="1.0">
	<entity name="account">
		<filter type="and">
			<condition attribute="statecode" operator="eq" value="0" />
		</filter>
	</entity>
</fetch>
<fetch version="1.0">
	<entity name="contact">
		<filter type="and">
			<condition attribute="statecode" operator="eq" value="0" />
			<condition attribute="statecode" entityname="L0" operator="ne" value="1" />
		</filter>
		<link-entity name="account" alias="L0" from="accountid" to="parentcustomerid" link-type="inner" />
	</entity>
</fetch>

IncSync fetches used by default take care only of changed records only. E.g., here is the contact fetch:

<fetch version="1.0">
	<entity name="contact">
		<filter>
			<condition attribute="versionnumber" operator="ge" value="13486549" />
		</filter>
	</entity>
</fetch>

The problem: When Server activates an existing account, IncSync will download that account, but will ignore its contacts. (Unless they were by chance modified, too.)

If you enable Linked SyncFilter for contacts, IncSync will use this fetch instead:

<fetch version="1.0">
	<entity name="contact">
		<filter type="or">
			<condition attribute="versionnumber" operator="ge" value="13486549" />
			<condition entityname="L0" attribute="versionnumber" operator="ge" value="13486549" />
		</filter>
		<link-entity name="account" from="accountid" to="parentcustomerid" link-type="outer" alias="L0" />
	</entity>
</fetch>

You see - we added an outer link to the changed accounts. In effect, all contacts belonging to accounts modified since the last sync will be downloaded, too. And that solves our problem. (Of course, at the cost of potentially increased downloads.)

Use case 2: Get records belonging to "my teams" (Dynamics only)

Here is the commonly used solution (where the employee entity has the attribute team that points to owning team):

<fetch version="1.0">
	<entity name="fwi_employee">
		<filter type="and">
			<condition operator="eq-userteams" attribute="fwi_team"/>
		</filter>
	</entity>
</fetch>

Not too bad, it works almost satisfactorily. I.e., unless the mobile user is added to or removed from another team. If we want a more reliable solution, we first have to understand team implementation.

The Dynamics CRM uses an NN relationship named teammembership that connects users and teams. This relationship is, in essence, another CRM entity (even though it is not displayed in Dynamics UI) which contains two lookups:

• systemuserid
• teamid

Hence, our solution will be as follows:

• Add teammembership entity to our customization.
• Define Employee sync filter.

<fetch version="1.0">
	<entity name="fwi_employee">
		<filter />
		<link-entity name="teammembership" from="teamid" link-type="inner" to="fwi_team">
			<filter type="and">
				<condition operator="eq-userid" attribute="systemuserid"/>
			</filter>
		</link-entity>
	</entity>
</fetch>

If we now switch on LinkedSyncFilters for Employee entity, SyncEngine will use this fetch for IncSync:

<fetch version="1.0">
	<entity name="fwi_employee">
		<filter type="or">
			<condition attribute="versionnumber" operator="ge" value="13486549" />
			<condition entityname="L0" attribute="versionnumber" operator="ge" value="13486549" />
		</filter>
		<link-entity name="teammembership" from="teamid" to="fwi_team" link-type="outer" alias="L0" />
	</entity>
</fetch>

This fetch downloads:

• All changed employee records
• All records belonging to teams with changed membership

This way, we solved the above problem. Mobile users will always have an up-to-date Employee table, even if the team regroups.

The cost is an increased (and possibly unneeded) download of Employee records. If we find this cost too high (and switch off LinkedSyncFilters attribute for Employee entity), but still want that the client reacts to team changes, we have to force FullSync after team changes.

Use case 3: LinkedSyncFilters for child entities

We saw before that FullSync automatically modifies sync filter for a child entity by adding inner link to parent entity (together with its sync filter).

Hence, the changes may come from two directions:

• From changing child records themselves, or
• From changing parent entity record (when it starts or stops matching sync filter)

This is how LinkedSyncFilter for a child entity is constructed:

<fetch version="1.0" aggregate="false">
	<entity name="quotedetail">
		<filter>
			<filter type="or">
				<condition attribute="versionnumber" operator="ge" value="759400043" />
				<condition entityname="quote_filter_0" attribute="versionnumber" operator="ge" value="759400043" />
			</filter>
		</filter>
		<link-entity name="quote" from="quoteid" to="quoteid" link-type="outer" alias="quote_filter_0" />
	</entity>
</fetch>

When to use linked sync filters

The LinkedSyncFilters option means the standard IncSync fetch receives one outer link for each linked entity used in the FullSync fetch. I.e., the mobile device will download:

• Records that have changed themselves
• Records that link to a changed record via one of the sync filter links
• Records that are child records of a changed parent.

The resulting fetches can be pretty complex and can cause not only large traffic, but also heavy server load.

Therefore, the SyncEngine tries to simplify them if possible. For example, if it knows that an entity does not have any changes (this information comes from SyncAnalyzer), the respective link is skipped.

We recommend that LinkedSyncFilters option should only be used if there is a reason, and only for selected entities.

Summary

Incremental sync is a powerful tool. You may run into problems, though.

• Server deletions - use Delete Plugin
• Newly shared/unshared records - use SyncShared
• Related records start matching the sync filter without being changed
• Moving time windows (time conditions with upper limit relative to the current date)

If IncSync has problems you cannot cure, think about the last possibility: Always Full Sync. This method is bulletproof in the sense that it never misses any record, whether it is a related unchanged record, a newly shared record, or a deleted record. And if your entity has only a few thousand client records, it might even have satisfactory performance.