Advanced sync setup

From Resco's Wiki
(Redirected from Sync Config)
Jump to: navigation, search
Synchronization

This document explains how to speed up the synchronization of databases between server and mobile devices.

  • We start by explaining the internals of the synchronization process.
  • Then we introduce main variables that control sync performance.
  • Finally, we'll explain how to formally write the setup string.

What is the SyncDownloader?

SyncDownloader executes perhaps the most important part of the synchronization - download of "normal" entity records.

  • It downloads accounts, contacts, orders, orderdetails...
  • It does not download some special entities such as NN entities or activityparty's.
  • It does not download attachments, images.
  • It does not execute upload of client changes to the server.
  • It does not perform sync with SharePoint, OneDrive, Google Drive etc.

For a typical configuration, most of the sync time is spent in the SyncDownloader.

Measuring performance

The following describes in deeper detail how the SyncDownloader works. If you understand this, you will be able to fine-tune the SyncDownloader to maximize its performance.

Let's start with a simple question: How can we measure the sync performance?

First of all, we shall compare FullSync times. (Incremental sync with its low data traffic and a relatively large overhead is difficult to judge.)

Secondly, we have two obvious metrics:

  • Total time spent in the download. You can find this number in the sync log and it looks like "...<FullSync> Entitys=54141ms..." (ms stands for milliseconds). This measure is good when we have a relatively stable set of server data. However, it does not allow comparing sync performance against different servers.
  • Alternative measure is the average time needed to download 1 record. Say we have this sync log excerpt:
<Summary ... Sent=0 Recv=366590 Result=Normal/> ... <FullSync> Entitys=54141ms...
The average time per record = 54141/366590 = 0.15 ms/rec. Speed ~1 ms/rec can be considered as a solid performance. Above number is much better and was achieved through the steps explained in the following.

How does SyncDownloader work

Entities are somehow sorted. The exact criteria are not important. The downloader opens (usually) 3 download threads that work simultaneously - hence the term multi-downloader. Each thread takes the first entity that was not yet downloaded and executes the download of all relevant entity records. When the download is ready, the downloader takes the next free entity. If there is no such entity, the download thread quits. The downloader finishes when all its download threads terminate.

The download of entity records is done in steps, also denoted as pages. By default, one page consists of (up to) 500 records. For example, if we need to download 1253 account records, the downloader issues 3 web requests that return subsequently 500, 500 and 253 records.

What happens to the downloaded records? The downloader has a cache, where the downloaded records are stored. All download threads write to the cache and the upper layer (SyncEngine) takes over the cached records and stores them in the database.

The cache has limited space and it often happens that it gets filled. Here is a typical scenario:

  • 3 download threads download subsequently accounts, contacts, and orders.
  • SyncEngine processes the account entity. (The account entity is active.)
  • Hence the account records do not remain in the cache (they are immediately consumed), but contacts/orders are stored in the cache at least until accounts download finishes.

So what happens when the cache gets filled? Of course, the download of inactive entities gets temporarily interrupted. (Will be resumed when the cache free space allows.) Download pauses slow down the overall performance. MCRM is rather conservative in allocating the cache space, hence pauses appear rather often, especially if downloaded entities have many records.

Parameters that control the download process

The exact usage of these parameters will be clarified later by examples. Here we provide just formal definitions.

NumDownloadThreads Count of download threads used by the downloader. Default=0, i.e. MCRM default applies. Allowed non-zero values: 3-10.
DownloadCacheSize In MB. Default=0, i.e. MCRM default applies. Allowed non-zero values: 10-4000 for Windows platforms, 10-1000 for mobile platforms.
DownloadPageSize Page size of download fetches. Default=0, i.e. MCRM default applies. Allowed non-zero values: 100-5000.

The last parameter that can be changed is the entity order. More precisely, the user can assign priority to selected entities. Such entities will be downloaded at the beginning, i.e. before other entities. You can do so for up to (NumDownloadThreads-1) entities.

MCRM defaults (as of MCRM 11.1):

  • NumDownloadThreads = 3. (Unless the app setting MultiThreadSync is switched off, in which case the SyncDownloader is not used. This is a very slow option - not recommended.)
  • DownloadCacheSize = 250 MB for Desktop/WinRT, 50 MB for WinUWP, 10-50 MB for Android/iOS (depending on the device parameters)
  • DownloadPageSize = 100 for emails, 500 otherwise

Typical problems

Before we discuss typical problems and try to formulate recommendations how to modify download parameters, let's explain how the client-server communication works. MCRM typically communicates over https. Here is a simplified web request life cycle:

  • Request composition. (Done instantly, which means the time spent in this step is negligible with respect to other steps.)
  • Sending request to the server. The duration - called latency - depends upon the connection bandwidth and the distance between the client and the server.
  • Server processing. This as a rule translates down to an SQL database query. Usually fast, but there are exceptions. Some queries may even timeout the server.
  • Sending server response to the client. This also counts to the latency.
  • Client processing. Includes response parsing and processing - usually writing to the client database. While these actions represent considerable effort, they typically represent only a fraction of the time spent in the web communication.

Web latency

Mobile networks are known for high latency. If you use a VPN, latency gets larger. And if your server is 10000 km away, latency grows even more. Under these circumstances, the whole web request life cycle may take well over 1 second. Theoretically, if the latency is 1.5 seconds and the 3 download threads run at the full speed (500 records per request), we get 1500 records in 1.5 seconds, i.e. 1 ms/record. However, this does not account for any overhead, for requests returning less than 500 records (they have the same latency!) and for download pauses. Hence, the real performance will be much smaller.

Recommendations:

  • Increase DownloadPageSize. Doubling the page size will halve the latency.
Risk: Higher memory consumption.
Tip: You may use different page sizes for different entities. If an entity has small records*), you can use a much higher page size.
  • Increase NumDownloadThreads.

*) "Small record" means record with small data. Records can be large either because of many nonempty columns (empty columns do not count) or because of large text column(s). Records containing a few hundreds of Bytes are considered small. Special case: Columns containing so-called attachments (activitymimeattachment.body, annotation.documentbody, salesliteratureitem.documentbody) are not counted. These columns use to contain big data and that's why they are excluded from SyncDownloader fetches. Instead, they are downloaded in a separate step as attachments.

Dominant entity

Imagine a model situation: 30 entities, 50000 contacts, remaining 29 entities have all together 50000 records as well. (This is what we call dominant entity.) How efficient is a multi-downloader in this situation? Let's do simple math: For simplicity, let the single thread download speed is 1000 records/second.

  • Single-thread downloader needs 100 seconds.
  • Multi-downloader (3 threads), contact entity is the last one: 67 seconds. (First 50000 records with the speed 3000 rec/sec, then 50000 records with the speed 1000 rec/sec.)
  • Multi-downloader (3 threads), contact entity is the first one: 50 seconds. (Download of all non-contact records is "for free".)

To complete the picture: If there was no dominant entity, then the theoretical multi-downloader speed would be 3000 rec/sec and it would need 33 seconds.

Recommendations:

  • Dominant entity: Set on priority so that the entity download starts right at the beginning.
  • Dominant entity: Increase the page size as high as possible. (Decreasing the latency)
  • NumDownloadThreads = 2. (Motivation: Remove every possible obstacle for the dominant entity. Probably a minor effect only.)

Platforms, devices

You can specify conditional setup valid for some platforms only; see the examples below. Here is the list of supported platforms (the names are case sensitive):

android Android devices
ios iPhone, iPad
Desktop, WinRT, WinUWP Windows platforms
Windows This is a shortcut for Desktop + WinRT + WinUWP
Mobiles This is a shortcut for android + ios

Another device-related condition is OnlyHighEndDevices='true'. As of MobileCrm 11.1 high end devices are:

  • All Windows devices,
  • iPhone5+, iPad3+,
  • Android: Devices with >100MB of available memory & able to allocate 50MB continuous memory block.
Note This definition might change in the future.

Editing the sync config file

To update your sync setup file, proceed as follows:

  1. Start Woodford.
  2. Edit an app project.
  3. Select Configuration from the Project menu.
  4. Click Sync Config.
  5. Edit the XML configuration file, then click Save.

We recommend that you prepare the file in an external editor and verify that it’s a valid XML file (for example, try opening the file in a web browser); then copy the content into Woodford.

Advanced sync setup.png

Alternatively, for advanced testing and troubleshooting on the Windows platform, you can directly edit the file MobileCrm\Configuration\SyncSetup.xml in the project directory. Once you are happy with the results, copy your setup into Woodford as described above, to make it available with all users of the project.

Sync log information

Advanced settings generally contain several platform-dependent setups. SyncEngine selects the settings applicable to the current platform. Moreover, settings may be justified to allowed ranges. To double-check which setup values are actually used in the synchronization, check the log from the last synchronization. Look for a text similar to:

<AdvancedSetup>
DownloadCacheSize=500
DownloadPageSize=2000
Entities: my_entity1[priority, DownloadPageSize=5000] my_entity2[DownloadPageSize=5000]
</AdvancedSetup>

This text is permanently stored in the log before the first use of the advanced settings. (For the case something goes wrong.) Sync log contains additional information that is important for designing downloader cache. It might look similar to <Downloader CacheSize=200MB UsedCache=50MB />. In this case, we designed an unnecessary large cache. This does not present any problem because the cache grows dynamically (DownloadCacheSize is just theoretical upper limit), yet it is good to know the real memory consumption.

The downloader summary may contain also the information about the pauses ,as for example <Downloader CacheSize=100MB UsedCache=81MB Paused=18% />. In this case, we know that the downloader paused for a considerable time. As the single possible reason is insufficient cache size, increasing the cache size would improve the performance.

Warning

Default MCRM settings are rather conservative to fit all supported devices. Millions of successful synchronizations prove this point. Here is how the settings evolved:

  • MCRM used DownloadPageSize=500 since the very beginning. (Except emails which used page size 20. Long emails presented a problem in those days.)
  • January 2013: MCRM introduced multi-downloader with 3 threads.
  • July 2016: DownloadCacheSize was increased for the last time.
  • June 2017: Page size for emails was moved to 100. (More precisely, EmailFetchPageSize app setting was introduced with the default value 100.)

Mobile devices improve every year, there is certainly room for using higher than the default values. However, keep in mind that by doing so you also increase the memory requirements on the device. Never use these settings in the production without thorough preliminary testing.

Examples

// 5 threads, 5000 records per page, 1GB cache size. Valid for all platforms.
// Should result in very fast sync - unless there will be a memory problem (mobile devices). Nevertheless, the setup should work for the desktop.
<SESetup Version='1'>
	<SyncDownloaderSetup>
		<Setup DownloadCacheSize='1000' DownloadPageSize='5000' NumDownloadThreads='5' />
	</SyncDownloaderSetup>
</SESetup>
// Desktop: 5000 records per page, 1GB cache size.
// High end devices: 5000 records per page, 300 MB cache size.
// Other devices: MCRM defaults apply
<SESetup Version='1'>
	<SyncDownloaderSetup>
		<Setup Platform='Desktop' DownloadCacheSize='1000' DownloadPageSize='5000' NumDownloadThreads='5' />
		<Setup DownloadCacheSize='300' DownloadPageSize='5000'  OnlyHighEndDevices='true'/>
	</SyncDownloaderSetup>
</SESetup>
// All Windows platforms: 5000 records per page, 1GB cache size.
// Mobile devices: MCRM defaults apply
<SESetup Version='1'>
	<SyncDownloaderSetup>
		<Setup Platform='Desktop WinRT WinUWP' DownloadCacheSize='1000' DownloadPageSize='5000' />
	</SyncDownloaderSetup>
</SESetup>
// Yet another example of platform-dependent setup.
<SESetup Version='1'>
	<SyncDownloaderSetup>
		<Setup Platform='Desktop WinRT WinUWP' NumDownloadThreads='5' DownloadCacheSize='1000' DownloadPageSize='5000' />
		<Setup Platform='android ios' DownloadCacheSize='1000' DownloadPageSize='5000' />
	</SyncDownloaderSetup>
</SESetup>
// Same effect as above
<SESetup Version='1'>
	<SyncDownloaderSetup>
		<Setup Platform='Desktop WinRT WinUWP' NumDownloadThreads='5' DownloadCacheSize='1000' DownloadPageSize='5000' />
		<Setup Platform='Mobiles' DownloadCacheSize='1000' DownloadPageSize='5000' />
	</SyncDownloaderSetup>
</SESetup>
// The new feature here is specifying priority for an entity.
// Multi-downloader will start its job by downloading this entity. (Suitable for a dominant entity.)
<SESetup Version='1'>
	<SyncDownloaderSetup>
		<Setup Platform='Desktop' DownloadCacheSize='1000' DownloadPageSize='5000'>
			<Entities>
				<Entity Name='mvow_serialnumber' HasPriority='true' />
			</Entities>
		</Setup>
	</SyncDownloaderSetup>
</SESetup>
// This complex setup illustrates how competing platform setups are evaluated.
// In this case there are two setups applicable to Desktop platform.
// The last one is selected as it provides the most specific platform specification.
<SESetup Version='1'>
	<SyncDownloaderSetup>
		<Setup Platform='Desktop WinRT WinUWP' DownloadCacheSize='1000'>
			<Entities>
				<Entity Name='mvow_serialnumber' DownloadPageSize='2000' HasPriority='true' />
				<Entity Name='mvow_accountaddress' DownloadPageSize='2000' />
				<Entity Name='mvow_itemnumber' DownloadPageSize='2000' />
			</Entities>
		</Setup>
		<Setup Platform='Mobiles' OnlyHighEndDevices='true'>
			<Entities>
				<Entity Name='mvow_serialnumber' DownloadPageSize='1000' HasPriority='true' />
				<Entity Name='mvow_accountaddress' DownloadPageSize='1000' />
				<Entity Name='mvow_itemnumber' />
			</Entities>
		</Setup>
		<Setup Platform='Desktop' DownloadCacheSize='1000' NumDownloadThreads='5'>
			<Entities>
				<Entity Name='mvow_serialnumber' DownloadPageSize='4000' HasPriority='true' />
				<Entity Name='mvow_accountaddress' DownloadPageSize='2000' />
				<Entity Name='mvow_itemnumber' DownloadPageSize='2000' />
			</Entities>
		</Setup>
	</SyncDownloaderSetup>
</SESetup>

Custom upload order

You can use advanced sync setup to modify the order, in which the client changes are uploaded to the server. This functionality was introduced in a bugfix release in January 2020 (release 12.3.1).

Default behavior

Let's start by explaining how MCRM uploads user changes.

  1. MCRM starts by collecting entities with changed records.
    Changed records are uploaded entity by entity. It cannot happen that, e.g. uploaded accounts would be intermixed with uploaded contacts.
  2. Next, MCRM decides about the upload order, i.e. which entity is uploaded first, which one comes next, etc. The decision is based on these goals:
    • Parent record (quote, invoice) are uploaded before child records (quotedetail, invoicedetail). See also note below.
    • Minimize forward references. In other words, the lookup target record should be uploaded before the lookup source record. (To avoid lookups pointing to records which were not uploaded yet.)
    • The following entities are uploaded at the very end: businessprocessflowinstance (entity used for business process modelling; Dynamics only) and resco_mobileaudit (entity storing MCRM audit information).

The upload order can be checked in the sync log. (You need to enable sync details in configuration of your app project.)

Note You can influence the upload order by defining child-parent relations. In Woodford, edit the app project, select the child entity from the Project menu, select the lookup that points to the parent record, and on the Properties pane check Mark as Parent.

In most scenarios, the default order is good enough. However, there are situations when an optimal upload order does not exist, typically when there are circular references between changed records. In these cases, MCRM attempts other strategies, for example, partial uploads. A detailed explanation is beyond the scope of this document.

Moreover, there are situations that cannot be solved on the client side, e.g. when the server uses a plugin that requires a specific order of uploaded entities.

Custom order

To address the problems related to the upload order MCRM allows specifying a custom upload order. Custom upload order consists of commands (instructions) that are applied after the upload entities were sorted by the default algorithm.

In particular, you can specify:

  • Which entities are uploaded at the beginning (FirstEntities) and which entities are uploaded at the end (LastEntities) of the upload process.
  • The order of two or more entities (Sequence).
  • Any number of statements of the type "Entity A should be uploaded before Entity B" (Relations).

If your custom commands contradict each other, some are silently ignored.

It is not necessary to define the order of all entities. Usually, it's better if you only set the order of those entities which you want to manage.

Technical details

The order of application of custom commands:

(1) FirstEntities
  • Only one <FirstEntities> element is allowed, but it may contain any number of entities.
(2) Sequence
  • Only one <Sequence> element is allowed, but it may contain any number of entities.
  • Unknown entities (or entities which do not have any changes) are eliminated; remaining Sequence elements (if any) are respected.
(3) Relations
  • Only one <Relations> element is allowed, but it may contain any number of relations.
  • Invalid relations (either source or target is an unknown entity) are ignored.
  • Remaining relations are interpreted one by one in a cycle.
  • The whole cycle is repeated several (but finite number of) times until all relations are fulfilled. (If possible.)
(4) LastEntities
  • Only one <LastEntities> element is allowed, but it may contain any number of entities.

Understand the difference between Sequence and Relations. For example:

Sequence: Entity1 < Entity2 < Entity3
Relations: Entity1 < Entity2, Entity2 < Entity3

It may look like the two rules above have the same effect, however, there is a difference in the transitivity. If Entity2 does not exist (or has no changes), then we are left with this:

Sequence: Entity1 < Entity3
Relations: - (nothing)

Another important difference between above rules is that Sequence can't express multi-child relations, e.g.

Relations: Entity1 < Entity3, Entity2 < Entity3

Examples of custom upload order

// Simple custom order, where we only specify that account entity should be uploaded at the very end after all other entities (if any) are uploaded.
// Note the syntax - <SyncDownloaderSetup/> is omitted. (This node is optional.)
<SESetup Version='1'>
	<UploadOrder>
		<LastEntities>
			<Entity>account</Entity>
		</LastEntities>
	</UploadOrder>
</SESetup>
// Complex example, where we specify which entities should be uploaded at the start, resp. at the end.
// Additionally, we use Sequence to specify sort order of some of the remaining entities.
<SESetup Version='1'>
	<SyncDownloaderSetup>
		<Setup Platform='Desktop WinRT WinUWP' DownloadCacheSize='1000' DownloadPageSize='5000' />
	</SyncDownloaderSetup>
	<UploadOrder>
		<FirstEntities>
			<Entity>entity8</Entity>
			<Entity>entity9</Entity>
		</FirstEntities>
		<Sequence>
			<Entity>entity7</Entity>
			<Entity>entity6</Entity>
			<Entity>entity5</Entity>
			<Entity>entity4</Entity>
			<Entity>entity3</Entity>
			<Entity>entity2</Entity>
		</Sequence>
		<LastEntities>
			<Entity>entity0</Entity>
			<Entity>entity1</Entity>
		</LastEntities>
	</UploadOrder>
</SESetup>
// Another relatively complex example.
// entityX denotes entity, which is omitted from the client customization.
// This is no error - MCRM simply ignores such entities. (This provides more flexibility when writing custom orders.)
<SESetup Version='1'>
	<SyncDownloaderSetup/>
	<UploadOrder>
		<Sequence>
			<Entity>entity8</Entity>
			<Entity>entity2</Entity>
			<Entity>entityX</Entity>
			<Entity>entityX</Entity>
			<Entity>entityX</Entity>
		</Sequence>
		<LastEntities>
			<Entity>entity1</Entity>
		</LastEntities>
	</UploadOrder>
</SESetup>
// Example illustrates a specific upload order, where we define the order of all entities.
// This is a risky setup (you have to analyze all lookups), but if you know what you are doing, the option is here.
<SESetup Version='1'>
	<SyncDownloaderSetup/>
	<UploadOrder>
		<Sequence>
			<Entity>entity9</Entity>
			<Entity>entity8</Entity>
			<Entity>entity7</Entity>
			<Entity>entityX</Entity>
			<Entity>entity6</Entity>
			<Entity>entity5</Entity>
			<Entity>entity4</Entity>
			<Entity>entity3</Entity>
			<Entity>entity2</Entity>
			<Entity>entity1</Entity>
			<Entity>entity0</Entity>
		</Sequence>
	</UploadOrder>
</SESetup>
// Example illustrating another possibility of how to influence the upload order:
// This time we specify that entity A should be uploaded prior to entity B plus a similar instruction for entity C and entity D.
// You can specify any number of such relations, just make sure they do not contradict each other.
<SESetup Version='1'>
	<SyncDownloaderSetup/>
	<UploadOrder>
		<Relations>
			<Relation Entity='entityA' Before='entityB' />
			<Relation Entity='entityC' Before='entityD' />
		</Relations>
	</UploadOrder>
</SESetup>
// Complex example, where we use several mechanisms to adjust the upload order.
<SESetup Version='1'>
	<SyncDownloaderSetup/>
	<UploadOrder>
		<Sequence>
			<Entity>entity7</Entity>
			<Entity>entity6</Entity>
			<Entity>entity3</Entity>
		</Sequence>
		<Relations>
			<Relation Entity='entity8' Before='entity0'/>
			<Relation Entity='entity9' Before='entity0'/>
			<Relation Entity='entity5' Before='entity4'/>
		</Relations>
		<LastEntities>
			<Entity>entity2</Entity>
		</LastEntities>
	</UploadOrder>
</SESetup>
// Last example is here to demonstrate one formal aspect: Order of XML nodes is not prescribed.
<SESetup Version='1'>
	<SyncDownloaderSetup/>
	<UploadOrder>
		<LastEntities>
			<Entity>entity2</Entity>
		</LastEntities>
		<Relations>
			<Relation Entity='entity0' Before='entity3'/>
			<Relation Entity='entity9' Before='entity0'/>
		</Relations>
		<Sequence>
			<Entity>entity7</Entity>
			<Entity>entity6</Entity>
		</Sequence>
	</UploadOrder>
</SESetup>

Sync log notation

Information about the used custom upload order can be found in the sync log in the <UploadOrder> node, as shown in the example below. Note that this information is omitted if there are no changes to be uploaded to the server.

SyncStart:2020-01-02 14:29:46+01:00

<UploadOrder>
    <FirstEntities> entity1 entity2</FirstEntities>
    <LastEntities> entity3</LastEntities>
</UploadOrder>
Date:2020-01-02 14:30:16+01:00
Org:...

If you enabled the diagnostic sync log via configuration, every single upload action is logged.

<UploadSync>

<Upload entity=contact>
1> Create contact[John Doe, f2d10927-51a6-4db4-9f90-1c9df524b59e]
MultiSend> 1 packets in 254ms
ClearLog f2d10927-51a6-4db4-9f90-1c9df524b59e
</Upload>

<Upload entity=resco_mobileaudit>
1> Create resco_mobileaudit[applicationstate, 5ceaf7df-b283-4cff-8e09-a9e79b9e3bf7]
2> Create resco_mobileaudit[synchronization, 4f20ab68-744d-41bb-9393-40adb057dd6a]
MultiSend> 2 packets in 844ms
ClearLog 5ceaf7df-b283-4cff-8e09-a9e79b9e3bf7
ClearLog 4f20ab68-744d-41bb-9393-40adb057dd6a
</Upload>

</UploadSync>

Upload:NoChangesLeft
...
<UploadSync> Entitys=969ms Sent=3 SvrCalls=1 TotalTim=969</UploadSync>

In the above example, we see this upload order: 1. contacts (1 record), 2. audits (2 records).