AI Image Recognition

Resco can integrate with Azure's Custom Vision service to classify images. This essentially allows functions similar to barcode scanning - except you don't need barcodes!

Prerequisites

Mind the following prerequisites for using this function:

• Azure Custom Vision license or subscription and at least one model for classifying images
• Connection to the Internet to access Azure servers that perform the necessary image analysis
• Proper configuration in Woodford (see below)

Custom Vision main steps

Azure Custom can be found here: https://www.customvision.ai/. For more information, visit https://azure.microsoft.com/en-in/services/cognitive-services/custom-vision-service/.

1. Follow this tutorial: https://docs.microsoft.com/en-in/azure/cognitive-services/custom-vision-service/getting-started-build-a-classifier#create-a-new-project
2. When creating a new project:
   • As Project Types select Classification.
   • As Classification Types select Multilabel.
3. After creating the project, go to the Performance tab, then click Prediction URL.
   
4. This displays the information necessary for Woodford configuration.
   

Change field format to barcode

The entity where you want to use image recognition has to include a field with barcode format.

1. Using Woodford, edit an app project and select the entity from the Project menu.
2. Select the field. It must have the type Single Line of Text.
3. On the Properties pane, set Formatting to Barcode.
4. Click Save.

For more information about managing entities and fields in Woodford, see Managing entities.

Set up connection to Azure services

1. Using Woodford, edit an app project and select Settings > AI Image Recognition from the Project menu.
2. Click New to define a new model.
3. As Name, enter the name of the configuration model.
4. As Entity, enter the entity where the model should be used.
5. As Service type, use Azure.
6. For Prediction key and URL, use the values from your Azure Custom Vision site.
7. Click OK to save the model.

You can define multiple models as needed.

Use case: Find a record in a view

For entities that include a barcode field and have at least one model created, there's an additional button in the views for image recognition. It allows you to take a photo. The picture is then analyzed by Azure. If the resulting tag (with the highest probability) equals the value of the field with barcode formatting, that record is selected. In effect, this is similar to barcode scanning, using an image tag instead of a barcode for record identification.

If a single entity has multiple fields with barcode format, the result of the scan is compared to all these fields.

It is also possible to use image recognition in view rules. The Scan Image step is available in On Save, On Change, Button Click, and Cell Click events. It is a good practice to add Scan Image inside the condition when used in On Change, Button Click, Cell Click rules - see the rule example in the forms use case below.

Be extra careful with On Change rules. Avoid rules like "if field1 has changed, scan image and save the result to field1". This leads to a loop.

Use case: Place image classifier to a field in a form

Using rules, you can add a custom function to your forms that initiates camera and image classification. If the image matches the model (with a probability above a customizable threshold), the name of the model is saved to a field.

For example, you can add a row button to your form that uses the Scan Image step to fill the value on a form.

1. Edit a form.
2. Click Add Button and set Binding to MyScanButton.
3. Click Button Click to open the rules editor.
4. Add the following rule and save your changes.

If	Condition	ChangedItem	Equals	MyScanButton
Then	Scan:	Action: AI Image Recognition	0.9 [1]	Save to: Entity.name	Probability: <Pick Value>

[1]: Specify minimum threshold between 0 (0% match) and 1 (100% match).

Results can be saved to a field or to a (shared) variable.

• If saved to a field or to a string variable, only the top result (with the greatest probability) is stored.
• If saved to a string list variable, all results in a form of a list are stored.

The same applies to probabilities. It does not make much sense to store one in a string and the other in a string list, but it is possible.

When you are using string lists to store the results, you can set a threshold (between 0 and 1). Only results with probabilities higher than the threshold are saved. Results are always ordered: the highest probability match comes first.

Use case: Scan image to answer questions in questionnaires

Using rules in Questionnaire Designer, you can add a custom function to your questionnaire template that initiates camera and image classification. If the image matches the model (with a probability above a customizable threshold), the name of the model is saved as an answer. This is available for all string-type answers.

See also

• Let the AI do the work Blog