[SEO Subhead]
This Guidance shows how to convert images to text and speech with machine learning and generative AI services on AWS. Converting images to text is done with the help of Amazon Kendra, a search engine that can be used to index an image repository and search for data. Next, generative AI is used for captioning the images, recognizing objects and features to generate a human-readable textual description, typically a caption based on extracted visual features. This Guidance also shows how to convert image to speech and can be extended to serve content through voice-enabled devices, such as Amazon Alexa. This involves the Describe for Me web app which generates a caption of an image and reads it back in a clear, human-sounding voice, including a variety of languages and dialects.
Please note: [Disclaimer]
Architecture Diagram
-
Image-To-Text
-
Image-To-Speech
-
Image-To-Text
-
This architecture diagram demonstrates how to convert image to text.
Click to enlarge
Step 1
Users upload images to an image repository like an Amazon Simple Storage Service (Amazon S3) bucket.Step 2
An AWS Lambda function is invoked to enrich the Amazon Kendra index on demand.Step 3
Amazon Kendra, a search engine that can be used to search for structured and unstructured data, indexes the image repository. The Lambda function invokes both Amazon Textract and the generative AI model that is deployed on Amazon SageMaker as a near real-time inference endpoint.Step 4
The generative AI model for image captioning processes the uploaded images, recognizing objects and features to generate a human-readable textual description, typically a caption based on extracted visual features.
Step 5
Simultaneously, another component of the solution utilizes Amazon Textract. Amazon Textract is used to extract any text that is visible within the image and is invoked by the Lambda function. This extracted text is also considered as part of the metadata for the image.
Step 6
The image captions are stored in an Amazon S3 bucket.
Step 7
Both the generated image description from the generative AI model and the extracted text from Amazon Textract are considered as metadata for the uploaded image.This metadata is associated with the image within the Amazon Kendra index.
Users can then search for images using natural language queries, such as “Find images of red roses” or “Show me pictures of dogs playing in the park,” through the Amazon Kendra console, an AWS software development kit (AWS SDK), or API. These queries are processed by Amazon Kendra, which uses machine learning (ML) algorithms to understand the meaning behind the queries and retrieve relevant images from the indexed repository.
Step 1
Users upload images to an image repository like an Amazon Simple Storage Service (Amazon S3) bucket.Step 2
An AWS Lambda function is invoked to enrich the Amazon Kendra index on demand.Step 3
Amazon Kendra, a search engine that can be used to search for structured and unstructured data, indexes the image repository. The Lambda function invokes both Amazon Textract and the generative AI model that is deployed on Amazon SageMaker as a near real-time inference endpoint.Step 4
The generative AI model for image captioning processes the uploaded images, recognizing objects and features to generate a human-readable textual description, typically a caption based on extracted visual features.
Step 5
Simultaneously, another component of the solution utilizes Amazon Textract. Amazon Textract is used to extract any text that is visible within the image and is invoked by the Lambda function. This extracted text is also considered as part of the metadata for the image.
Step 6
The image captions are stored in an Amazon S3 bucket.
Step 7
Both the generated image description from the generative AI model and the extracted text from Amazon Textract are considered as metadata for the uploaded image.This metadata is associated with the image within the Amazon Kendra index.
Users can then search for images using natural language queries, such as “Find images of red roses” or “Show me pictures of dogs playing in the park,” through the Amazon Kendra console, an AWS software development kit (AWS SDK), or API. These queries are processed by Amazon Kendra, which uses machine learning (ML) algorithms to understand the meaning behind the queries and retrieve relevant images from the indexed repository.
-
Image-To-Speech
-
This architecture diagram demonstrates how to convert an image to speech; it can be extended by serving the content through voice-enabled devices, such as Alexa.
Click to enlarge
Step 1
An AWS SDK connects the mobile client to AWS Amplify.Step 2
The Amazon Cognito identity pool grants users temporary access to the Amazon S3 bucket.Step 3
A user can upload an image file to the Amazon S3 bucket.
Step 4
The Describe for Me web app invokes the backend artificial intelligence (AI) services by sending the Amazon S3 object key in the payload to Amazon API Gateway.Step 5
API Gateway instantiates an AWS Step Functions workflow. The state machine uses Lambda functions to orchestrate the AI/ML services Amazon Textract, Amazon Rekognition, Amazon SageMaker, Amazon Translate, and Amazon Polly.Step 6
The Step Functions workflow creates an audio file output and stores it in Amazon S3 in MP3 format.
Step 7
A presigned URL with the location of the audio file stored in Amazon S3 is sent back to the user’s browser through API Gateway. The user’s mobile device plays the audio file using the presigned URL.
Step 1
An AWS SDK connects the mobile client to AWS Amplify.Step 2
The Amazon Cognito identity pool grants users temporary access to the Amazon S3 bucket.Step 3
A user can upload an image file to the Amazon S3 bucket.
Step 4
The Describe for Me web app invokes the backend artificial intelligence (AI) services by sending the Amazon S3 object key in the payload to Amazon API Gateway.Step 5
API Gateway instantiates an AWS Step Functions workflow. The state machine uses Lambda functions to orchestrate the AI/ML services Amazon Textract, Amazon Rekognition, Amazon SageMaker, Amazon Translate, and Amazon Polly.Step 6
The Step Functions workflow creates an audio file output and stores it in Amazon S3 in MP3 format.
Step 7
A presigned URL with the location of the audio file stored in Amazon S3 is sent back to the user’s browser through API Gateway. The user’s mobile device plays the audio file using the presigned URL.
Well-Architected Pillars
The AWS Well-Architected Framework helps you understand the pros and cons of the decisions you make when building systems in the cloud. The six pillars of the Framework allow you to learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems. Using the AWS Well-Architected Tool, available at no charge in the AWS Management Console, you can review your workloads against these best practices by answering a set of questions for each pillar.
The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.
-
Operational ExcellenceRead the Operational Excellence whitepaper
This Guidance uses AWS services like Lambda and Step Functions to automate tasks, reducing manual work and errors, and Amazon S3 to provide reliable data storage. These services easily adapt to changing workloads and support efficient, consistent operations. Additionally, you can use Amazon CloudWatch to monitor operations and gain insights.
-
SecurityRead the Security whitepaper
This Guidance uses Lambda and Step Functions to automate security-related tasks, reducing the risk of human error in security processes. Additionally, API Gateway enforces secure management of API endpoints, Amazon Cognito enhances user authentication and authorization processes, and AWS Identity and Access Management (IAM) controls access to AWS resources. Finally, CloudWatch helps detect security incidents or anomalous activities in real time, facilitating swift incident responses and threat mitigation.
-
ReliabilityRead the Reliability whitepaper
This Guidance uses automation through Lambda and Step Functions to reduce the risk of human errors that might compromise reliability. Additionally, Amazon S3 provides data replication and redundancy features that increase data reliability, and API Gateway grants users consistent and secure access to APIs to maintain workload reliability. CloudWatch monitors operations, aiding in issue detection and resolution. This proactive approach enhances workload reliability by minimizing downtime and disruptions.
-
Performance EfficiencyRead the Performance Efficiency whitepaper
This Guidance reduces latency and resource inefficiency by using Lambda and Step Functions to automate processes and streamline workflows. Additionally, SageMaker and Amazon Polly facilitate real-time content generation, supporting faster and more efficient workloads, and API Gateway optimizes API management, delivering low latency and consistent access to promote high performance efficiency.
-
Cost OptimizationRead the Cost Optimization whitepaper
This Guidance minimizes operational expenses by using Lambda and Step Functions to facilitate efficient resource use and reduce the need for constant manual intervention, minimizing human error and resource waste. Additionally, Amazon Polly reduces the need for costly manual content creation, API Gateway optimizes API management, decreasing compute-related costs, and Amazon Kendra improves search efficiency, reducing the time and resources spent on information retrieval. Finally, Amazon S3 offers scalable and cost-effective storage solutions so that you can store and access data efficiently without incurring unnecessary expenses.
-
SustainabilityRead the Sustainability whitepaper
This Guidance uses serverless services like Lambda and API Gateway for their energy efficiency, their efficient use of resources, and their incorporation of renewable energy sources. These practices align with sustainability goals, helping you reduce your carbon footprint.
Implementation Resources
A detailed guide is provided to experiment and use within your AWS account. Each stage of building the Guidance, including deployment, usage, and cleanup, is examined to prepare it for deployment.
The sample code is a starting point. It is industry validated, prescriptive but not definitive, and a peek under the hood to help you begin.
Related Content
Automate caption creation and search for images at enterprise scale using generative AI and Amazon Kendra
Introducing an image-to-speech Generative AI application using Amazon SageMaker and Hugging Face
Disclaimer
The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.
References to third-party services or organizations in this Guidance do not imply an endorsement, sponsorship, or affiliation between Amazon or AWS and the third party. Guidance from AWS is a technical starting point, and you can customize your integration with third-party services when you deploy the architecture.