JAWS Picture Smart AI

 

JAWS is a powerful accessibility solution that reads information on your screen using synthesized speech. JAWS provides many useful commands that make it easier to use programs, edit documents, and read Web pages.

JAWS® is the world's most popular screen reader, developed for computer users whose vision loss prevents them from seeing screen content. JAWS reads aloud what's on the PC screen and gives the user a unique set of intelligent tools for navigating and accesses Web pages and all screen content. 

Now, even "inaccessible" images and diagrams can now be described efficiently with the amazing "Picture Smart AI" utility.

While navigating the digital landscape, you might encounter photos, charts, and other visual elements without an alternative description. This can make it challenging to understand the information provided by the visual content. The JAWS® Picture Smart AI feature allows you to submit a photo to be analyzed and then displays a description of the picture in the JAWS Results Viewer window.

This revolutionary update to the original Picture Smart feature, introduced in 2019, integrates with the latest artificial intelligence (AI) technology from services like Chat GPT from Open AI and Claude from Anthropic to offer a whole new level of accessibility to graphical information. Descriptions might offer details such as the number of people in the photo including how they look and what activities they may be performing, recognizable landmarks, landscape details such as the color of the sky, grass, or trees, the layout of information in a visual chart or slide, and much more. Additionally, any text in the picture such as writing on a sign or piece of clothing may also be provided in the description.

Once a submitted image finishes processing, Picture Smart AI displays a short description by default. To display more in-depth descriptions, select the Get more details link near the bottom of the Results Viewer. You can also add SHIFT to a Picture Smart command to immediately display the more detailed descriptions. For example, INSERT+SPACEBAR, P, SHIFT+F.

For a list of Picture Smart keyboard commands, click here.

To demonstrate the difference in the level of detail, here are a few examples along with resulting descriptions generated using both the original Picture Smart and Picture Smart AI. You can also try this feature out yourself by pressing INSERT+SPACEBAR, P, C while focused on each photo.

Current Picture Smart Sample Description Picture Smart AI Sample Short Description Picture Smart AI Sample Long Description
Caption is timeline. These tags describe the photo: display, font, screenshot, text. These tags probably describe the photo: diagram, number. The image is a timeline that spans from September 2024 to May 2025. It is divided into 5 sections, each with a different task and deadline. The tasks are as follows:

– September 2024: Synergize scalable e-commerce
– November 2024: Disseminate standardized metrics
– January 2025: Coordinate e-business applications
– March 2025: Foster holistically superior methodologies
– May 2025: Deploy strategic networks with compelling e-business needs		 The image you provided is of a slide with a title “TIMELINE” at the top center. Below the title is a horizontal timeline graphic that spans from September 2024 to May 2025. The timeline is divided into five sections, each with a designated month and year, and an associated objective:

1. “SEP 2024” with the objective “Synergize scalable e-commerce”
2. “NOV 2024” with the objective “Disseminate standardized metrics”
3. “JAN 2025” with the objective “Coordinate e-business applications”
4. “MAR 2025” with the objective “Foster holistically superior methodologies”
5. “MAY 2025” with the objective “Deploy strategic networks with compelling e-business needs”

Each section is represented by an arrow pointing to the right that starts from the month and extends to the right with the objective contained in the body of the arrow. The colors of the image are mostly shades of blue and grey, with the timeline graphic against a lighter background. At the bottom right corner, there’s a page number or slide number indicated by “11,” and just above it, the year “2024” is written as a note. The overall aesthetic is professional and likely part of a business presentation.


Asking Questions About a Picture

By popular request, Picture Smart AI offers the ability to send follow-up prompts or questions in order to obtain additional details about a picture that may not have been covered in the initial description.

Once you have performed a Picture Smart AI command such as INSERT+SPACEBAR, P, F, and the description is displayed in the Results Viewer, press ENTER on the Ask questions about this image link near the bottom of the window. If you chose the option to get more details or included SHIFT as part of the command to view detailed descriptions from both AI services, then the link is called Ask Chat GPT questions about this image.

After selecting the link to ask questions, you are immediately placed in an edit box. Type a prompt and press ENTER. For instance, for a picture of a beach at sunset, you might say, “can you describe the sunset and how it looks going down over the ocean.” After a few seconds, during which JAWS will play a sound to indicate progress, the Results Viewer window updates with the response, and JAWS automatically reads it. Your focus remains in the edit box, so you can type another prompt.

In situations where you already have some basic knowledge about an image, you can include ALT with any Picture Smart command to immediately ask a question about specific details.

For example, suppose you were shopping on Amazon for a product like an Multicooker or Air Fryer. You already have a basic idea of what the product is, but you may want to know upfront if the product contains physical buttons or a touch activated display. You could press INSERT+SPACEBAR, P, ALT+C, type your question followed by ENTER, and Picture Smart AI attempts to answer the question about the selected image instead of just providing a general description. Once the results containing the response are displayed, you can still use the Ask questions about this image link to ask follow-up questions if needed.

Your questions and responses are maintained for the current Results Viewer session until you close the window. To review and copy specific responses for pasting in another application, press ESC to exit Forms Mode, and then use H or SHIFT+H to navigate to the heading that corresponds to the question and response you want to copy.

Following is an example picture to demonstrate a question and response.

The image displays an organizational chart with various positions and names. The chart is headed by a President, followed by an Executive Assistant and several Vice Presidents (VPs) overseeing departments such as Finance, Technology, Operations, Marketing, and Production. Under the VPs are different managerial and specialist roles like Accounting, Help Desk, Facilities, Project Manager, Production Manager, and Dispatch. The chart uses color-coding to differentiate between national, domestic, and international roles.

User Question:

Who is marked as international?

Response:

In the organizational chart, the “Transportation VP” role, held by “Ian Hansson,” is color-coded as international. This is indicated by the blue color of the box associated with that position.

 

Summary of Picture Smart commands

To use Picture Smart, press INSERT+SPACEBAR followed by P to access the Picture Smart layer, and then use one of the following keystrokes:

  • A – Opens the Picture Smart dialog box where you can acquire an image from a flatbed scanner or the Freedom Scientific PEARL camera and submit it to be described.
  • F – Describes the currently selected image in File Explorer. Supported image formats include BMP, JPEG, GIF, TIF, PNG, or HEIC.
  • C – Describes any currently selected control. For instance, if focused on  a picture in a Word document or on a social media website like Facebook, this command describes the picture. When using this command on a web page, make sure your browser window is maximized for best results .
  • B – Describes the contents of the Windows Clipboard if it contains an image.
  • S – Describes the entire screen.
  • W – Describes the current application window.
  • ENTER – Describes an image based on the current application and context. This eliminates the need to remember multiple keystrokes constantly. For instance:
    • If someone is presenting in Teams, it describes the current slide.
    • If you are on a web page, it describes the current image or control.
    • If you are in File Explorer, it describes the currently selected file.
    • If playing a video from a source like YouTube, it describes a screenshot from the current position in the video.
    • If a PowerPoint slideshow is running, it describes the current slide.
    • If you are in PowerPoint with a slide selected but not in an active slideshow, it describes the currently visible slide.
  • QUESTION MARK – Displays help for the Picture Smart layer.