OCR turns a recorded graphical session into text you can search. Instead of watching hours of video to find the moment someone opened a particular file or console, you search for the words that were on screen and jump straight to that point in the recording.
Command logs work well for terminals: you can search every command typed. They say nothing about a Windows desktop. An administrator on RDP can open Notepad, a browser tab, Active Directory Users and Computers or a cloud console, and a command log records none of it. Recording captures it, but a recording nobody can search is a recording nobody watches.
How OCR session search works
- Capture. The session is recorded as video, together with a transcript for terminal sessions.
- Sample. Frames are taken at an interval. A sample every couple of seconds balances cost against the chance of missing something short-lived.
- Recognise. An OCR engine reads the text in each frame. Open-source engines such as Tesseract can run entirely on your own servers.
- Index with timestamps. Every recognised line is stored with its offset into the recording and put into a full-text index, with near-duplicate lines collapsed so a static window does not produce thousands of identical hits.
- Search. A query returns matching recordings and the exact moments, and opening a result starts playback at that offset.
For terminal sessions the transcript is indexed directly, so shell and graphical sessions can be searched from the same place.
What you can find
- Incident scoping. "Which sessions had
payrollopen?" in seconds, instead of reviewing every session on that server. - Audit sampling. An auditor asks for sessions where a production database console was visible, and you produce them with timestamps.
- Investigation. After an alert, find the moment a particular tool, URL or file name appeared on screen.
- Policy checks. Look for applications or consoles a session should never have shown.
Turning search into detection
Once on-screen text is indexed, it can feed more than a search box:
- Threat enrichment. Deterministic keyword and pattern detectors run over the text and raise risk events that link to the moment in the recording.
- Intent review. Compare the on-screen applications and commands against what the session was approved to do, and flag differences after the session.
- Live checks (opt-in). Running OCR on live sessions allows a warning or termination when a denied application appears. Because the text is recognised after it has been painted, this is a response, not a substitute for blocking at the host.
Prefer deterministic matching for these detectors. A keyword rule that says "this exact term appeared" is explainable and testable. A model that guesses a screen is suspicious is neither.
The privacy problem you create
OCR produces a second copy of everything that was on the screen, in searchable form. That has consequences you must design for:
- Redact secrets before indexing. A password, token or private key visible in a frame must not end up in a search index that more people can query than the recording itself. Redaction should happen at ingest, and cover multi-line material such as private-key blocks.
- Control who can search. Search access should follow the same permissions as playback, and be audited.
- Apply retention. Index entries should expire with the recording they came from.
- Keep it on-premises. Sending screenshots to a cloud vision service to get text is exactly the data egress privileged-access tooling exists to prevent. On-premises recognition keeps frames inside your boundary.
Accuracy: what to test
OCR is text recognition, not understanding, and quality depends on the session:
| Factor | Effect | Test |
|---|---|---|
| Font size and display scaling | Small text is read less reliably | Record at your real resolution and zoom |
| Blur and compression | Smears characters | Use your actual recording settings |
| Language and scripts | Engines are tuned for specific languages | Test the languages your admins use |
| Dark themes and low contrast | Fewer correct reads | Include your actual themes |
| Sampling interval | Text visible for under a sample interval can be missed | Check short-lived dialogs |
| Terminal vs graphical | Terminal transcripts are exact; graphical OCR is probabilistic | Treat them as different confidence levels |
Run a pilot with your own recordings: plant known words (a file name, a console title, a hostname) in a handful of sessions, search for them, and measure how many are found. A vendor's accuracy figure on a demo video tells you little about your environment.
A pilot checklist
- Record three sessions with a known script: open a text editor and type a unique string, browse to an internal console, open a file manager showing specific file names.
- Search for each planted term and note whether it is found and whether the timestamp lands on the right moment.
- Display a fake secret on screen and confirm it is not retrievable from the search index.
- Check that a user without audit permissions cannot search.
- Confirm indexing finishes in a reasonable time after a session ends, and that a failed index run is visible, not silent.
- Confirm nothing leaves the deployment during indexing.
How ZenXPAM supports this
ZenXPAM's recording text search uses on-premises Tesseract OCR on graphical sessions, sampled every two seconds by default, and indexes terminal transcripts alongside it. Every line is stored with its timestamp in a full-text index, and the dedicated search page filters by user, resource, date range and source and opens the player at the matching moment. Secrets visible on screen, including private-key text, are redacted before indexing. Keyword and pattern detectors raise risk events that seek to the moment in the recording, on-screen text can be compared with the session's approved scope after the session, and live OCR with warn or terminate responses is available as an opt-in. No screenshots leave the deployment and no cloud vision service is used. OCR search is a newer capability, so test it on your own recordings as described above. See threat detection and investigation or book a demo.
FAQ
Is OCR the same as AI vision? No. OCR recognises text. It does not interpret what is happening on screen. That is a strength for auditability: a text match is simple to explain.
Can OCR search replace watching recordings? It makes watching targeted. You still watch the part that matters.
Does it work for terminal sessions? Terminal sessions are better served by their transcript, which is exact. OCR is for graphical sessions where there is no transcript.
What about non-English text? Recognition quality depends on the engine and the language data installed. Test the languages your administrators actually use.
Will it slow down sessions? Indexing runs after the session on recorded video, so the live session is unaffected. Live OCR is optional and uses resources, so cap its concurrency.
