Documents
| Integration with third-party systems |
|---|
|
Resco CRM Connector (web APIs):
|
If you need to store a large number of documents, you don't have to rely on CRM attachments. Resco Mobile CRM can leverage cloud storage services, such as OneDrive or SharePoint.
Introduction
When talking about documents or attachments, we refer to a rather large group of file types: PDF, Microsoft Office documents (Word, Excel, PowerPoint, etc.), HTML files, images, video files...
In a CRM, documents are traditionally stored as attachments on CRM servers. However, Resco supports also files stored on external document servers and we refer to them as documents.
Resco supports 5 document servers - Box, DropBox, GoogleDrive, OneDrive, and SharePoint; users may use any number of them.
The basic principles are the same as for attachments:
- A document must be attached to a single existing CRM record.
- A CRM record can have multiple attached documents.
- Standalone documents are unsupported. A document becomes orphaned if its CRM record gets deleted. Orphaned documents are deleted in sync cleanup, or by cascading delete.
The integration to document servers must be enabled in the app, and it must be also configured in the Woodford project.
Log in to your document server in the app
If your document servers are properly configured (see below), you need to log in to your document server to start using it.
- Start Resco Mobile CRM or other Resco app that supports documents.
- Tap Setup on the home screen.
- In the Accounts section, log in to the document server by using your credentials (or the credentials of your company).
- Save your setup.
- Return to the home screen and synchronize your app.
| Note | All documents that you want to access must be downloaded to the offline database, even if you want to use Mobile CRM app in online mode. |
Box, Dropbox, Google Drive, OneDrive
- Edit an app project in Woodford.
- Select Documents from the Project menu.
- Check Enable Documents and click Save.
- Select Configuration from the Project menu and go to Integrations tab.
- Enable the document storage that you're using and specify the root folder, then click Save.
Your documents are saved to the following directory structure:root\entity_name\record_name_record_id. - Next, edit the forms for various entities and add Cloud Document List to the entity form, so that users can access documents associated with entity records.
| Warning | Integration with the document server may fail if the root folder does not exist. Verify that the root exists; if it does not, create it manually. |
Out of the box, the integration with SharePoint document storage is only available on Microsoft Dynamics CRM and Resco Cloud.
- Configure your backend server (Resco Cloud or Dynamics).
- Configure your Woodford app project.
Server-side prerequisites
Resco mobile apps support both SharePoint List Component base integration and Server-Side Integration. Both will result in the creation of the required records. The difference is in the CRM web user interface, which is not important for Resco Mobile CRM. The List Component is deprecated, but please read the following guidelines to make sure you pick the right choice.
See SharePoint documentation depending on your installation type:
Resco Mobile CRM requires that sharepointsite and documentLocation records are correctly configured. This means that these records must exist:
- At least one sharepointsite record with the SharePoint URL
- At least one documentLocation per SharePoint-enabled-entity root
Configure Resco Cloud
To enable SharePoint in Resco Cloud, perform the following settings in the Admin Console.
- Select Data > Manage Data from the menu, then click Show System Entities.
- Go to the SharePoint Site entity and click New to create a new record.
- Click Show System Fields to access all fields for the record.
- Enter a Name and the Absolute URL parameter, enable Default Site, then click Save.
- Go to the Document Location entity and click New to create a new record for each entity for which you want to enable SharePoint.
- Enter a Name and as Parent Site or Location select the site created in step 2, and as Relative URL, enter the logical name of the entity. Click Save.
- Enter a Name and as Parent Site or Location select the site created in step 2, and as Relative URL, enter the logical name of the entity. Click Save.
The settings used in this example allow adding documents for account records. The Resco Mobile CRM app creates a Document Location record for each account record with a SharePoint document. If you want to create such records manually in advance:
- Set Relative URL to {account_name}_{Guid} (for example, Apollo Appartments_3EB67D9478414AAAAD06E06AD8D15C2B).
- Set Regarding to 'Apollo Appartments' (lookup for account record).
- Set Parent Site or Location to 'Accounts Docs' (lookup to the document location record created in the previous step).
An additional configuration step is necessary directly on your SharePoint server: Create a document library with the name equal to the logical entity name. If such a document library does not exist, synchronization of documents fails.
You can now enable SharePoint in Woodford as described below.
Configure Dynamics
| Warning | Don't create these records manually! Use the wizard instead. |
- In Dynamics CRM, go to Settings > System > Document Management.
- Click Document Management Settings to start the wizard.
- Select entities that should have SharePoint documents integration.
- Enter the URL of the SharePoint site.
- Optionally, select the Folder Structure and Automatic folder creation.
- Click Finish to conclude the wizard.
- Edit an app project in Woodford.
- Select Documents from the Project menu.
- Check Enable Documents and Enable SharePoint.
- Enabling SharePoint integration will also enable SharePoint Site entity in the mobile project. Please do not make any changes to this entity.
- If you have SharePoint on premise, you can define the internal and external SharePoint server URL.
- Click Save.
- Select Configuration from the Project menu and go to Integrations tab.
- Scroll down to SharePoint section and select Enable SharePoint.
- Enable one of the APIs, then click Save.
- Use Partial SharePoint REST API (legacy) if you don't use OAuth2 or if you're working with a very large number of SharePoint lists. The legacy interface is also suitable for SharePoint on premises.
- Use SharePoint REST API (OAuth2) for SharePoint Online with OAuth2 authorization shared with Dynamics (single sign-on). This option also supports multi-factor authentication.
- Next, edit the forms for various entities and add Cloud Document List to the entity form, so that users can access documents associated with entity records.
- Finally, if your login credentials for SharePoint differ from your CRM credentials, go to Mobile CRM app's Setup and log in to SharePoint. If SharePoint uses different authentication or you use CRM on-premise vs SharePoint online (or vice versa) than CRM, you also need to specify the Type.
Troubleshooting
SharePoint documents can only be used when the documents and associated records are available in the offline database (even if you are using the app in online mode). The records that you are using in online mode must be synchronized. Adjust Sync Filters for specific entities to ensure that the necessary records are downloaded and available in offline mode.
Document Location records need to be downloaded to the offline database. They are needed (among other things) to define the folder structure, i.e., the path to SharePoint server documents. Clear the entire Sync Filter for the Document Location entity. If the documents become available in the app, but you need to limit the Document Location records, continue with the next step.
The document location defining the CRM entity root folder must have its regarding set to NULL (empty). Otherwise, they are treated as file location records; this breaks the folder structure used in the app. If you need to filter Document Location entity records, make sure that records with an empty regarding are OR-ed with the other conditions; for example:
SharePoint is organized via sites and lists. Resco mobile apps synchronize with specific SharePoint lists - document libraries.
- Each document location represents a subfolder of some document library.
- Documents represent document library list items.
In the Dynamics CRM world, Resco relies on existing Dynamics SharePoint integration consisting of two entities:
- sharepointsite - lists SP sites used in the integration
- sharepointdocumentlocation - Lists document locations, i.e. folders of document libraries and their corresponding CRM records.
In the standard integration, CRM admin selects entities (account, contact…) that can have SharePoint attachments. In turn, respective SharePoint document libraries are created: One library per one entity with the library names identical to the entity names.
Resco mobile apps sync first these two above-mentioned entities (as part of the standard sync) and then use them to control communication with SharePoint.
Make sure that these entities exist and are enabled in your backend. In the case of Salesforce, the backend admin probably needs to create them manually. Then you can enable the entities in your app project in Woodford. At the minimum, the sharepointsite table must have a single record pointing to a suitable SharePoint site; document locations are created by Resco apps when needed.
Resco apps support 3 alternative ways of SharePoint communication:
- The oldest ListService interface is used by default. Needs extra API calls to obtain necessary info about document libraries. (GetListCollection does not return root folders.)
- Enable Partial SharePoint REST API (legacy) configuration setting to use ListService partially using REST API. This interface is more efficient than the first because it does not need GetList API calls.
- Enable SharePoint REST API (OAuth2) configuration setting to use the full REST API. It is just as efficient at the second option. Additionally, it uses the OAuth authorization, with a single sign-on with Dynamics possible.
External document synchronization filter options
Document filters available in the Configuration section of Woodford allow you to exclude specific documents from download to the Mobile CRM app’s database, based on the file type, entity, a file they’re associated to, how many and what maximum size per file type, etc.
See also this part of our webinar. Webinar
Cloud storage for questionnaire images
Since release 14.0, you can now store questionnaire images in various cloud document storage solutions such as SharePoint or Google Drive, rather than in the CRM database. This applies both to the static images (such as questionnaire logo) or images that inspectors take when answering image-type questions. Storing images outside of the database can be important from a cost perspective.
Configuration in Woodford
External document storage must be enabled and properly configured in Woodford.
- Edit an app project.
- Select Settings > Documents from the Project menu, check Enable Documents and optionally Enable SharePoint, then click Save.
- Select Settings > Configuration from the Project menu, go to the Integrations tab, enter the root folder for your images, then click Save.
- Publish the app project.
Usage in the Questionnaire Designer
The storage is configured for each questionnaire image component in the Storage Entity property.
Dynamic images are automatically saved to the configured location. For static images, add the image in the Questionnaire Designer, note down the name and location where the app should look for the image, and copy the image with the correct name to the configured cloud storage and path.
If you want to use SharePoint storage with Resco Cloud backend, you need to create a Document Location record for the questionnaire entity using the Admin Console as described here.
Technical information
On the device, external documents are stored in two locations:
- Document headers are stored in the database entity resco_sharepointdoc. Each record describes one document - its file name, path, size... but mainly the lookup to the owning CRM record (resco_regardingid).
- Document bodies are stored separately in the blob storage.
Document synchronization
Documents are synchronized in a separate step after the sync of CRM records is completed.
The download is organized similarly to attachments:
- Download of document headers from external document server. If you have multiple servers, they are queried one by one. The result of this step is an updated resco_sharepointdoc table containing headers of (non-downloaded) documents.
- Not all known documents need to be downloaded. Woodford admin may use either the Max Attachment Size setting or the BlobStoreFilter to determine which documents are downloaded. Non-downloaded documents can be downloaded on demand from UI. Documents that should be downloaded are enqueued in a so-called FileDownloadQueue.
- The download queue works in parallel to the above step. Its task is to download the document bodies (files). If the download is interrupted, the queue resumes its work at the next sync. Unlike attachments, asynchronous document download is unsupported.
The upload phase is conceptually simpler: Document changes (creations/deletions) are uploaded one by one.
Caveats
Documents may consume a large portion of device memory.
- Use MaxAttachmentSize setting or BlobStoreFilter to limit needed storage.
- To measure the used document storage (need to be done on Windows) go to the %APP_DATA% folder, then subfolder blob, then subfolder resco_sharepointdoc.
API call consumption
- Some calls are needed for document headers. (Exact specification is server-dependent.)
- Each downloaded document requires 1 call.
Dynamics SharePoint integration
- Dynamics lately added another SP-related entity: sharepointdocument - used internally by Dynamics for presentation purposes.
- Woodford admins might be tempted to enable this entity, but this would be pure overhead - Resco apps do not use it.
- (Moreover, for MCRM versions prior to v12.2 this entity causes sync failure. (Because it does not have the rowversion field.))
Too many GetList calls
- Applies to SharePoint with old ListService API. If you use SharePoint for many purposes, you might end up with 10K+ lists and the sync might get penalized by long delays (up to several hours per sync!). While Resco apps offer some tricks for this problem, the simplest and best solution is to start using the REST API. This eliminates GetList calls altogether.
Root document locations
- Document locations are hierarchical. Resco apps require that you have a root location for each entity type which allows SharePoint attachments. For example, you need to have a location for "accounts". If the location exists, MCRM can create a location for any particular account record. It does so by creating a subfolder with the name <record_name>_<record_id>.
- In the opposite case, Resco Mobile CRM refuses to create SharePoint attachments for the account entity.
- Only documents located directly in the folder of a CRM record are downloaded. Resco Mobile CRM does not support additional folder structure; documents in subfolders are not downloaded.
Enterprise-level document storage considerations
When it comes to the storage of image data to third parties, Resco currently has support for the following approaches, both of them used by several customers on enterprise-scale implementations.
- Server-side integration
Store images in Notes (annotation) tables with (optional) server-side solution for transparent offloading blobs to more cost-efficient storage. Some of our customers successfully use the Jenkins plugin, but other viable alternatives probably exist. This is the most flexible approach as it is transparent to Resco. You take advantage of the full power of our sync engine optimizations, sync filters with links to other tables, document filters, and server-side functionality such as triggers.
The main advantage is that you have all images as first-class citizens in the server database, and they can be naturally linked to other records. This approach also works with web-based Resco tools aimed at back-office users (e.g., browser version of questionnaire player). Generally speaking, this is what we would normally recommend as the most robust solution.
- Client-side integration
Resco mobile platform natively supports integration with various cloud-based services. The users of Resco mobile apps can attach blobs to records, but behind the scenes, documents are not stored in CRM but in the cloud service. It is important to note that it is the mobile app that provides the experience of meshing CRM data with cloud blobs, and there is no explicit linking between records and blobs in CRM. Resco uses a predefined folder structure to store blobs in cloud services and uses a naming convention to locate the files (with SharePoint, you have some more control over blob locations).
With this approach, you can take advantage of an optimized sync engine and document filters, but on the other hand, it is not possible to finetune sync to such an extent (e.g., no links to other tables in sync filters). Also, web-based Resco tools (e.g., questionnaire player in browser) don’t support cloud documents. There are some more limitations and caveats, most notably that cloud documents require parent records to be in the offline database to function properly. So to sum it up, this is what we would recommend as a very robust and convenient option in case you don’t need your blobs in CRM and accept the other limitations mentioned above.
- Outlook for 2023 and beyond
Since 2019, Microsoft's preferred way of storing blobs in Dataverse and CRM Online is File/Image columns. Behind the scenes (although the curtain is rather thin), these are stored in Azure Storage and count against File capacity. Since release 16.0, Resco offers basic support of the File column (read-only mode on entity forms, manual download+cache). Contact your business development manager at Resco if you're interested in learning about (and shaping!) Resco's plans in this area.
- Disclaimers
- We at Resco are experts on mobility and data synchronization with Dynamics/Dataverse. We don’t specialize in server-side implementations, the quality of third-party server tools, and operational cost optimization. That is the domain of our partners, who have much more experience and thus can provide more insight and recommendations.
- The analysis above is written with Dataverse/Dynamics ecosystem in mind, but similar conclusions also apply to other backends.
Additional resources
For more information about documents:
- Watch this webinar about integration of external documents. Webinar
- Read a series of technical deep dive articles: part 1, part 2, part 3, part 4 Blog