Computer Vision: Find Text Using Regex

The article describes some of the rules around locating text on the screen using the Computer Vision text locator.

When using Computer Vision to find text on the screen, the AI Employee reads the region one line at a time, starting from the top, and stops at the first line that matches your search. Within that line, it uses the fewest words possible that still match, and that is where it clicks or reads. A later line is never chosen over an earlier one, even if it would match with fewer words.

For instance, suppose the AI Employee searches for “this text” in the following block of text.

Line 1: this is some text on the screen.
Line 2: find this text in a sentence.
Line 3: find this text here.

The AI Employee will return the location of the words “this text” from Line 2, because it is the first line that matches.

Line 1 does not match because the words “this text” are not consecutive in Line 1.
Line 2 contains “this text” consecutively, so this is a match, and the search stops here.
Line 3 also contains “this text”, but the AI Employee never gets to it, because Line 2 matched first.

(Note: We are not using a regular expression (regex) in the example search above. We are just searching for two consecutive words.)

Whole words come first

When you search for plain text, the AI Employee looks first for that text as a whole word, anywhere in the region. Plain text means a search with none of these characters in it: . * + ? ^ $ ( ) [ ] { } | \

For example, a search for Price clicks a label that reads “Price:” rather than a button that reads “Update SalesPrice”, even when the button comes first on the screen. Only when the text doesn’t appear as a whole word anywhere does the AI Employee accept it inside a longer word. Before it does, it reads the region again more carefully, in case it missed the standalone word the first time.

Searches written as regular expressions aren’t affected by this rule.

Now, if we search for the regular expression “this.*text”, all three lines match it, so the AI Employee returns the location of the words “this is some text” from Line 1.

Line 1 matches the regex, and it is the first line, so the search stops here. The fewest words on Line 1 that still match are “this is some text” (4 words).
Line 2 would match with fewer words (“this text”, 2 words), but the AI Employee never gets to it, because Line 1 matched first.
Line 3 is never checked either.

To find the “this text” on Line 2 instead, make the search more specific, so that Line 1 no longer matches. For example, the plain search “this text” (the first example above) skips Line 1, because those two words are not next to each other there.

Now, if we want to return the location of the rest of the line instead of just the specific words that match the regex, we can anchor the search using regex syntax (^ anchors to the beginning of the line and $ anchors to the end of the line).

Using our above example, the regex “this text.*$” will return the location of the words “this text in a sentence.” from Line 2, because the regex asks for the consecutive words “this text” followed by everything up to the end of the line (specified by the $ anchor).

Line 1 does not match the regex because the words “this text” are not consecutive in Line 1.
Line 2 matches the regex, and it is the first line that does, so the search stops here. The match runs from “this” to the end of the line: this text in a sentence. (5 words).
Line 3 would also match, with fewer words (this text here), but the AI Employee never gets to it, because Line 2 matched first.

To return the whole line from its very first word, anchor both ends: “^.*this text.*$” returns all of Line 2, find this text in a sentence.

To reach Line 3, search for something only Line 3 contains, such as “this text here”.