Skip to content
ProcessPlanProcessPlan Docs

Computer Vision: Find Text Using Regex

The article describes some of the rules around locating text on the screen using the Computer Vision text locator.

When using Computer Vision to find text on the screen, the system will locate the first matching text on the screen that satisfies the search criteria with the fewest words possible.

For instance, suppose the AI Employee searches for “this text” in the following block of text.

Line 1: this is some text on the screen.
Line 2: find this text in a sentence.
Line 3: find this text here.

The AI Employee will return the location of the words “this text” from Line 2 because it is the first line found on the screen that satisfies the search and there are no other lines that satisfy the search with fewer words.

  • Line 1 does not match because the words “this text” are not consecutive in Line 1.

  • Line 2 contains “this text” consecutively, so this is a match.

  • Line 3 also contains “this text” consecutively, however Line 2 is the first match found and Line 3 does not match with fewer words.

(Note: We are not using a regular expression (regex) in the example search above. We are just searching for two consecutive words.)

Now, if we search for the regular expression “this.*text” the system will still return the location of the words “this text” from Line 2. Even though all three lines technically match this regex.

  • Line 1 matches the regex with “this is some text” (4 words)

  • Line 2 matches the regex with fewer words (“this text” = 2 words)

  • Line 3 also matches the regex with 2 words; however, Line 2 is the first match found and no other lines match with fewer words.

Now, if we want to return the location of the entire line instead of just the specific words that match the regex, then we can anchor the search using regex syntax (^ anchors to the beginning of the line and $ anchors to the end of the line).

Using our above example, the regex “this text.*$” will return the location of the words “this text here” from Line 3 because the regex states that we want to find the consecutive words “this text” followed by any arbitrary number of characters until the end of the line (specified by the $ anchor).

  • Line 1 does not match the regex because the words “this text” are not consecutive in Line 1.

  • Line 2 does match the regex. However, the match contains five words (this text in a sentence).

  • Line 3 matches the regex with only three words (this text here), so its location is used and returned.