1

Storage analyzer

From Resco's Wiki
Jump to navigation Jump to search
Resco Support

Storage analyzer is a collection of tools providing statistical data about the local storage of Resco mobile apps, with a particular focus on database storage and binary documents (blob storage).

Warning The reports include tips and recommendations, however, they are primarily aimed at technical readers. Consider consulting with your technical support provider or Resco support before performing any configuration changes in production.

Prerequisites

  • Storage analyzer requires app version 14.1. Some reports may require a later version, see below for details.
  • Storage analyzer is not available on Resco web apps (because they don't have any storage).

Using storage analyzer

Storage analyzer UI in Resco mobile apps
  1. Go to the Setup/Settings screen, scroll down to the Advanced section, and select StorageAnalyzer.
  2. Tap Select and choose one of the available reports, or choose all of them.
  3. Wait until the report is generated.
  4. Optionally, you can email the generated report; or use the email preview screen to copy text from the reports.

If you have a large database, generating the reports can take significant time. Tap Abort to cancel the report generation.

To clear the results pane, tap Clear Results.

AppData report

This report provides general guidance about the data consumption of Resco mobile apps. Other reports provide more details, especially about the database and blob storage. 

	Customization = 3074K
	WWW (Custom JS) = 2839K
	Images (Custom images) = 177K
	DB size = 6936M
	DB log size = 0K
	BlobStore = 1224M
	Logs size = 38K
	Temp files = 1309K
	Other files = 57K
	Temp Customization = 734K
	 ---> Totally 8168M
  • Customization – refers to the folder that contains all Woodford files created by Resco.
  • WWW (Custom JS) – JavaScript Woodford files supplied by the customer.
  • Images (Custom images) – Woodford images supplied by the customer.
  • DB size – SQLite database file size (represents entity records downloaded from the server)
  • DB log size – SQLite WAL log file size (temporary database storage responsible for guaranteeing data integrity in extreme situations, such as power outage)
  • BlobStore – Folder containing data stored outside of the database in the file system, i.e., attachments and documents.
  • Logs – Various log files maintained by Resco mobile apps. Use the About form to preview their content.
  • Temp files – Temporary files. (Mainly customization *.zip files.)
  • Other files – Remaining files maintained by Resco mobile apps.
  • Temp Customization – Folder with new customization which failed to be applied for any reason - maybe app crashed or was forcefully killed. Will be cleared sooner or later.

Entities report

This report provides basic information about all tables stored in the database. 

    ======= Nonempty Tables =======
        
        campaign: 2722 rows, 43 cols, 390 By/row, 1038K (+indexes ~331K)
               column xx_desctriptionofexposures = 277K
               Empty columns:  entityimageid stageid emailaddress entityimage promotioncodename processid pricelistidTarget pricelistid
        email: 3 rows, 16 cols, 679 By/row, 2K (+indexes ~1K)
               Empty columns:  actualdurationminutes scheduledend scheduledstart
        resco_mobilereport: 3 rows, 11 cols, 20126 By/row, 59K (+indexes ~1K)
        resco_question: 4741142 rows, 38 cols, 348 By/row, 1614703K (+indexes ~443K) -> !!
               Empty columns:  resco_icon resco_answerstorage resco_defaultvalue
        resco_questiongroup: 1185001 rows, 21 cols, 1264 By/row, 1463507K (+indexes ~382K) -> !!
               column resco_rules = 1223101K
               Empty columns:  resco_description

Total size of data stored in DB = 986M
Note:
              DB overhead adds ~40%.
              The rest are table indexes - often >50%.
              (Internal tables use to take negligible space.)

    ======= Empty synced entities =======

        campaignitem
        contract
        contractdetail
        discount
        discounttype
        incident
        invoice
        …
        (By removing these entities from customization you could save … API call(s) in each sync) -> !!

In the example above, the campaign entity has 2722 records, 43 columns, and the table record stores on average 390 data bytes. The total size of the data stored in the campaign table is 1038K. Table indexes take up an estimated 331K.

The size occupied by individual table columns is listed only in specific cases:

  • If the column takes an important portion of the table space. (Dominant columns such as campaign.xx_desctriptionofexposures, resco_questiongroup.resco_rules in the example.)
  • If the column is empty.

Empty tables (tables containing 0 records) are reported together.

The report shows the net size of the data stored in the database, which is often less than ½ of the size of the database file. This is because the database has substantial overhead and also because of indexes that serve for fast data access.

Tips to consider

  • Do you need empty table columns? They cause a small penalty both in local database operations and in server fetches.
  • Do you need empty entities? Each such entity needs 1 fetch during synchronization.
  • Label "-> !!" marks tables whose analysis took a long time (right now "too long" means 1+ second). The reason is usually too many rows or too many columns. (Or an older, slower device.)
  • For tables containing too many records: Consider decreasing the row count by designing a stronger sync filter.

BlobStore report

The term BlobStore denotes file storage used for

  • Attachments
  • Documents
  • Entity images

All these items represent potentially large data and are thus stored outside of the database as separate files. 

	======= BlobStore =======

	product(images): 1801 file(s), 5080K
	annotation: 3694 file(s), 1144M
	Cloud docs: 281 file(s), 71M
	salesliteratureitem: 18 file(s), 3644K
	---> 1224M totally

	16 docs enqueued for download when sync resumes ( SharePoint*3 Box*13 )

	56 attachments enqueued for download when sync resumes (112M)

Explanation:

  • Entity product stores entity images in the BlobStore. A lot of small files, nothing important.
  • 3694 attachments (file annotations) take more than 1 GB. Worth investigating.
  • 281 small cloud documents, nothing important.
  • 18 salesliteratureitem records with an average 200K size.
  • The last two lines may appear if the last sync was interrupted before all attachments/documents could be downloaded.

Tips to consider

This report provides a summary of the downloaded items. For example, attachments that were not downloaded because of interrupted sync or because of size limit are not considered here. Other reports provide more info, especially about the attachments and cloud documents.

File annotations report

This report provides detailed information about:

  • Entities using annotation attachments,
  • Type of data stored in annotations as well as most important file size stats,
  • Similar stats are provided also for not downloaded files (if any).
		======= File Annotations =======

	Annotation counts by related entity:
		3 x account
		124 x campaign
		1 x msdyn_workorder
		7 x xx_contract
		414 x xx_contractmonitoring
		106 x xx_document
		1 x resco_chatpost
		1303 x resco_questionnaire
		1736 x salesorder
		---> 3695 Notes totally

		======= Analysis: All Notes =======

	Downloaded 3694 out of 3695

	Jpeg: 278x, AVG=2671K, MAX=7321K, Top10%Avg=4210K
		Consider downscaling Jpeg images -> !!
	PNG: 2743x, AVG=24K, MAX=125K, Top10%Avg=69K
	Reports: 435x, AVG=451K, MAX=466K, Top10%Avg=457K
	PDF: 224x, AVG=754K, MAX=6035K, Top10%Avg=3646K
	Others: 14x, AVG=77K, MAX=384K
	NotDownloaded: 1x, AVG=0K

		======= Analysis: Notes.resco_questionnaire =======

	Downloaded 1303 out of 1303

	Jpeg: 263x, AVG=2687K, MAX=7321K, Top10%Avg=4240K
		Consider downscaling Jpeg images -> !!
	PNG: 1040x, AVG=40K, MAX=125K, Top10%Avg=79K

		======= Analysis: Notes.salesorder =======
	Downloaded 1735 out of 1736

	Jpeg: 11x, AVG=2379K, MAX=2830K
		Consider downscaling Jpeg images -> !!
	PNG: 1283x, AVG=11K, MAX=111K, Top10%Avg=30K
	Reports: 432x, AVG=451K, MAX=466K, Top10%Avg=457K
	PDF: 1x, AVG=172K
	Others: 8x, AVG=115K, MAX=384K
	NotDownloaded: 1x, AVG=0K

Explanation:

  • Attachments are primarily used by questionnaires (1303) and sales orders (1736).
  • In general, attachments are small PNG images (2743).
  • There are 10x fewer Jpeg images, however, they occupy 10x more space (278*2671 > 700M).
  • Even the analyzer found this suspicious and showed warning "Consider downscaling Jpeg images".
  • Section "Analysis: All Notes" talks about all attachments.
  • Section "Analysis: Notes.resco_questionnaire" talks only about attachments related to questionnaires.
  • The row Jpeg: 263x, AVG=2687K, MAX=7321K, Top10%Avg=4240K means:
  • 263 jpeg images
  • Average file size = 2687K (2.6M)
  • Maximum file size = 7321K (7.1M)
  • Average file size of the largest 10% (highest decile) is 4240K (4.1M)

Tips to consider

  • Look for information on which entities and which file types consume most storage.
  • Analyze whether it would be useful to decrease either attachment count or their size.
  • For example 7MB jpeg images typically use resolution >10MPix and can be often downscaled by as much as 10-50x. Unless you have a specific reason, consider downscaling.

Cloud documents report

This report resembles the annotations report, i.e. it provides detailed information about:

  • Entities using documents
  • Document types used
  • The most important file size stats

Similar stats are provided also for not downloaded files (if any). 

	    ======= Cloud Docs =======

	Docs counts by related entity:
		15 x xx_device_location
		419 x xx_vertriebsmediathek
		8 x opportunity
		---> 442 Cloud docs totally

		======= Analysis: All CloudDocs =======

	Downloaded 438 out of 442

	Jpeg: 221x, AVG=241K, MAX=2143K, Top10%Avg=567K
		Consider downscaling Jpeg images -> !!
	PNG: 194x, AVG=53K, MAX=304K, Top10%Avg=145K
	Others: 23x, AVG=4435K, MAX=15M
	NotDownloaded: 4x, AVG=871K, MAX=3432K

		======= Analysis: CloudDocs.xx_vertriebsmediathek =======

	Downloaded 419 out of 419

	Jpeg: 212x, AVG=242K, MAX=2143K, Top10%Avg=564K
	   Consider downscaling Jpeg images -> !!
	PNG: 192x, AVG=52K, MAX=146K, Top10%Avg=136K
	Others: 15x, AVG=6800K, MAX=15M

Explanation:

  • Documents are primarily used by xx_vertriebsmediathek entity (419).
  • In general, most documents are images - either PNG (194) or Jpeg (221).
  • Jpeg images are substantially larger than PNG images.
  • Even the analyzer found this suspicious and showed warning "Consider downscaling Jpeg images".
  • For the meaning of statistical terms (AVG, MAX, Top10%Avg) see the description of FileAnnotationsReport.

LastSyncTime report

Synchronization (among other things) attempts to download server changes for all entities in the client customization. This is done through a set of fetches executed within some time period. Given the fact that these fetches cannot be executed at once, different entities are becoming up-to-date at different times.

This is what this report depicts. For example, the next example shows that the entity account became synced 13 seconds sooner than the entity annotation.

For those who want to dig deeper: The report shows also rowversions which are unlike sync times fully reliable and which should be read as follows: The client imported all account records with rowversion<=561666201. 

    ======= LastSyncTimeReport =======

    account  20-05-19 13:34:52, Ver=561666201
    annotation  20-05-19 13:35:05, Ver=561668382
    appointment  20-05-19 13:35:04, Ver=561668310
    businessunit  20-05-19 13:34:51, Ver=557155673
    campaign  20-05-19 13:34:52, Ver=561666226
    campaignitem  20-05-19 13:39:29, Ver=561711822

	Incomplete Entities -> !!
		contact

Tips to consider

  • If the last sync was aborted, you can encounter a big difference in sync times.
  • This report may list also "Incomplete Entities". The sync of these entities was interrupted prematurely because the limit Max Sync Records was reached.

DocuSign report

This report makes sense if you use electronic signatures from DocuSign. It lists technical details about the documents waiting to be signed. This kind of information can be useful for Resco support when investigating problems. 

       Sign jobs:
              SignPending (LocalId: 3b5bf584-2b1d-45e3-a15e-7e71267532db) for [annotation, Signature.png] -> [resco_questionnaire, Q2 Approval]
              DownloadPending (ServerId: 8da6a51f-f1e9-420d-a4ed-fb06b72cddb6) for SharePoint doc img_210312_103233.jpg -> [opportunity, Standort #2]

Explanation

Document signing is a complex process consisting of several steps:

  • Document is uploaded to the signing server.
  • Document is signed by the server.
  • Signed document is downloaded to the client, where it replaces the original document.

In the above example:

  • Signature.png stored in an annotation record attached to a questionnaire: Signing process initiated.
  • Snapshot attached to an opportunity record: Signed file waiting on the server for download.

Database internals report

A rather technical report showing the details of the database engine used. 

======= DbInternalsReport =======

LibVersion= Sqlite 3.8.8.1
DbSize= 7102060K
WalSize= 97844K
RegisterCollation() -> OK
Selected PRAGMAs:
    freelist_count= 6005
    page_size= 4096
    cache_size= 20000 pages
    fullfsync= 0
CompileTimeOptions: ENABLE_COLUMN_METADATA HAS_CODEC SYSTEM_MALLOC THREADSAFE=1 SQLITE_THREADSAFE=1

*** MemoryInfo ***
SQLITE_STATUS_MEMORY_USED=87882288/87892888
…

Explanation:

  • WalSize refers to the size of the database log file.
  • RegisterCollation reports an eventual error when NOCASE collation (case-insensitive sorting) is registered with SQLite engine. (“OK” is the expected result.)
  • See SQLite documentation for more details.

SharePoint

Requires Resco Mobile CRM version 15.1 or later. This report gives an overview of document location records. In addition to the basic stats, the report may contain various warnings/tips as shown in the following example. 

======= Analysis of sharepointdocumentlocation entity =======

sharepointdocumentlocation entity contains enabled fields that are not required. Extra fields may slow down sync, especially for large data sets.  -> !!

Root locations (sharepointdocumentlocation records):
    Documents at default site 1[4e7ba4ce-1234-e911-a97d-000d3a34a0aa] /appointment
    Documents at default site 1[c5d1fa02-1234-e911-a981-000d3a34a0aa] /opportunity
    Documents at default site 1[72c8eeae-1234-e911-a986-000d3a34a0aa] /contact
    Documents at default site 1[5b1dc708-1234-e911-a979-000d3a34a14b] /competitor
    Documents at default site 1[ace73084-1234-e911-a979-000d3a34a14b] /ext_deliverypricelist
    Documents at default site 1[48630f4b-1234-e911-a976-000d3a34ae50] /ext_otheractivities
    Documents at default site 1[06bc6cc2-1234-e911-a977-000d3a34ae50] /account
    Documents at default site 1[31b16a5b-1234-e911-a979-000d3a34ae50] /lead
    Documents at default site 1[74adfea6-1234-e911-a979-000d3a34ae50] /ext_businessopportunity
    Documents at default site 1[84c84969-1234-e911-a979-000d3a34afa9] /ext_phonecall
    Documents at default site 1[3992a56b-1234-eb11-a813-000d3a3be299] /product

Counts of locations pointing to CRM RECORDS IN OFFLINE (LOCAL) DB:
    account * 129
    appointment * 14106
    competitor * 5
    contact * 2
    ext_deliverypricelist * 1
    ext_otheractivities * 5914
    product * 2
---> 20159 totally
(Tip: Looks like too many locations synced / files downloaded. Check if they are necessary.)

Counts of locations pointing to CRM RECORDS NOT IN OFFLINE (LOCAL) DB:
    account * 1947
    appointment * 166192
    contact * 86
    ext_phonecall * 2 (entity not in the customization)
    ext_businessopportunity * 18
    ext_otheractivities * 63854
    lead * 646
    opportunity * 121
---> 232866 totally
(Tip: These records increase sync overhead. Remove them via better SyncFilter.)

Explanation:

  • Extra entity fields occupy unneeded storage and slow down sync, especially if a large number of locations are used. Woodford by default selects all required sharepointdocumentlocation fields; no other field is needed.
  • Every CRM entity using SharePoint documents needs to have defined so-called root location.
  • Every CRM record with SharePoint documents has one or more locations, where documents are stored on SharePoint.
  • FullSync queries SharePoint in batches of 100 locations/call. A high number of locations means, a) many server requests, and b) potentially too many files downloaded. The case shown in the example looks like overkill, hence the warning.
  • Unknown locations point either to entities not included in the customization or to records that were not downloaded to the client, i.e. excluded by the sync filter. The recommendation (tip) says that the sync filter for sharepointdocumentlocation entity should be adjusted to the customization to avoid unnecessary sync overhead.

Questionnaire statistics

This report introduced in release 17.0 offers more insights into the questionnaires and their local storage consumption in the Resco Mobile CRM. The report includes:

  • Number of templates and data questionnaires (answers), as well as their storage needs
  • Template statistics by storage type, template dependency, versioning
  • Largest data questionnaires
  • Largest templates (used/unused)
  • Largest columns for templates/data questionnaires
  • SyncDownloader stats highlights

See also

For a statistical overview of server storage (Resco Cloud), visit the Admin Console.




Was this information helpful? How can we improve?
Retrieved from "https://docs.resco.net/mediawiki/index.php?title=Storage_analyzer&oldid=10300"