+ Start a Discussion
Jason FlammangJason Flammang 

Upload binary stream to Amazon S3 using REST API PUT operation

I have a need to upload binary stream PDF files to Amazon S3.  I've seen the sample code available to use the REST API with the POST operation on visualforce page, however, I need to upload the file via APEX without user involvment, as I'm retrieiving the files from another database via their SOAP API.

I'm trying to do this using the PUT operation, but I don't think I'm doing the authentication correctly as I'm getting a 403 Forbidden response.

Any ideas?
 
public void uploadPDF(String binaryPdfString, String key, String secret){
        String Date = Datetime.now().formatGMT('EEE,   dd MMM yyyy HH:mm:ss z');
        String bucketname = 'BucketName';
        String method = 'PUT';
        String filename = 'FileName';
        HttpRequest req = new HttpRequest();
        req.setMethod(method);
        req.setHeader('Host','s3-us-west-2.amazonaws.com');
        req.setEndpoint('https://s3-us-west-2.amazonaws.com' + '/'+ bucketname + '/' + filename);
        req.setHeader('Content-Length', string.valueOf(binaryPdfString.length()));
        req.setHeader('Content-Encoding', 'base64');
        req.setHeader('Content-Type', 'pdf');
        req.setHeader('Date', Date);

        //get signature string
        String stringToSign = 'PUT\n\n\n'+formattedDateString+'\n\n/'+bucketname+'/'+filename;
        String encodedStringToSign = EncodingUtil.urlEncode(stringToSign,'UTF-8');
        String signed = createSignature(encodedStringToSign,secret);
        String authHeader = 'AWS' + ' ' + key + ':' + signed;
        req.setHeader('Authorization',authHeader);
        req.setBody(binaryPdfString);
        Http http = new Http();

        try {
            //Execute web service call
            HTTPResponse res = http.send(req);
            System.debug('RESPONSE STRING: ' + res.toString());
            System.debug('RESPONSE STATUS: '+res.getStatus());
            System.debug('STATUS_CODE: '+res.getStatusCode());

        } catch(System.CalloutException e) {
            system.debug('AWS Service Callout Exception: ' + e.getMessage());
        }

}

public string createSignature(string canonicalBuffer,String secret){
        string sig;
        Blob mac = Crypto.generateMac('HMacSHA1', blob.valueof(canonicalBuffer),blob.valueof(secret));
        sig = EncodingUtil.base64Encode(mac);

        return sig;

}

 
Best Answer chosen by Jason Flammang
Jason FlammangJason Flammang
well I guess I should have waited a bit longer to post this question...haha

turns out my signing string didn't need to be UTF-8 encoded (see old code "String encodedStringToSign = EncodingUtil.urlEncode(stringToSign,'UTF-8');"  )

Amazon's doucmentation mentions "The string to sign (verb, headers, resource) must be UTF-8 encoded", however I removed this piece and just ran my createSignature class using stringToSign (not UTF-8 encoded) and it worked!!

Also, it turns out that you can decode the binary stream and use the decoded Blob as the body of the response.  Otherwise, Amazon just displays the binary stream text on screen.

Here is my final code

Hope this helps someone else!
public void uploadPDF(String binaryPdfString, String key, String secret){
        String Date = Datetime.now().formatGMT('EEE,   dd MMM yyyy HH:mm:ss z');
        String bucketname = 'BucketName';
        String method = 'PUT';
        String filename = 'FileName';
        HttpRequest req = new HttpRequest();
        req.setMethod(method);
        req.setHeader('Host','s3-us-west-2.amazonaws.com');
        req.setEndpoint('https://s3-us-west-2.amazonaws.com' + '/'+ bucketname + '/' + filename);
        req.setHeader('Content-Length', string.valueOf(binaryPdfString.length()));
        req.setHeader('Content-Encoding', 'base64');
        req.setHeader('Content-Type', 'pdf');
        req.setHeader('Date', Date);

        //get signature string
        String stringToSign = 'PUT\n\n\n'+formattedDateString+'\n\n/'+bucketname+'/'+filename;
        String signed = createSignature(stringToSign,secret);
        String authHeader = 'AWS' + ' ' + key + ':' + signed;
        req.setHeader('Authorization',authHeader);
        Blob PDF = EncodingUtil.base64Decode(binaryPdfString);
        req.setBodyAsBlob(PDF);
        Http http = new Http();

        try {
            //Execute web service call
            HTTPResponse res = http.send(req);
            System.debug('RESPONSE STRING: ' + res.toString());
            System.debug('RESPONSE STATUS: '+res.getStatus());
            System.debug('STATUS_CODE: '+res.getStatusCode());

        } catch(System.CalloutException e) {
            system.debug('AWS Service Callout Exception: ' + e.getMessage());
        }

}

public string createSignature(string canonicalBuffer,String secret){
        string sig;
        Blob mac = Crypto.generateMac('HMACSHA1', blob.valueof(canonicalBuffer),blob.valueof(secret));
        sig = EncodingUtil.base64Encode(mac);

        return sig;

}

 

All Answers

Jason FlammangJason Flammang
well I guess I should have waited a bit longer to post this question...haha

turns out my signing string didn't need to be UTF-8 encoded (see old code "String encodedStringToSign = EncodingUtil.urlEncode(stringToSign,'UTF-8');"  )

Amazon's doucmentation mentions "The string to sign (verb, headers, resource) must be UTF-8 encoded", however I removed this piece and just ran my createSignature class using stringToSign (not UTF-8 encoded) and it worked!!

Also, it turns out that you can decode the binary stream and use the decoded Blob as the body of the response.  Otherwise, Amazon just displays the binary stream text on screen.

Here is my final code

Hope this helps someone else!
public void uploadPDF(String binaryPdfString, String key, String secret){
        String Date = Datetime.now().formatGMT('EEE,   dd MMM yyyy HH:mm:ss z');
        String bucketname = 'BucketName';
        String method = 'PUT';
        String filename = 'FileName';
        HttpRequest req = new HttpRequest();
        req.setMethod(method);
        req.setHeader('Host','s3-us-west-2.amazonaws.com');
        req.setEndpoint('https://s3-us-west-2.amazonaws.com' + '/'+ bucketname + '/' + filename);
        req.setHeader('Content-Length', string.valueOf(binaryPdfString.length()));
        req.setHeader('Content-Encoding', 'base64');
        req.setHeader('Content-Type', 'pdf');
        req.setHeader('Date', Date);

        //get signature string
        String stringToSign = 'PUT\n\n\n'+formattedDateString+'\n\n/'+bucketname+'/'+filename;
        String signed = createSignature(stringToSign,secret);
        String authHeader = 'AWS' + ' ' + key + ':' + signed;
        req.setHeader('Authorization',authHeader);
        Blob PDF = EncodingUtil.base64Decode(binaryPdfString);
        req.setBodyAsBlob(PDF);
        Http http = new Http();

        try {
            //Execute web service call
            HTTPResponse res = http.send(req);
            System.debug('RESPONSE STRING: ' + res.toString());
            System.debug('RESPONSE STATUS: '+res.getStatus());
            System.debug('STATUS_CODE: '+res.getStatusCode());

        } catch(System.CalloutException e) {
            system.debug('AWS Service Callout Exception: ' + e.getMessage());
        }

}

public string createSignature(string canonicalBuffer,String secret){
        string sig;
        Blob mac = Crypto.generateMac('HMACSHA1', blob.valueof(canonicalBuffer),blob.valueof(secret));
        sig = EncodingUtil.base64Encode(mac);

        return sig;

}

 
This was selected as the best answer
Jason FlammangJason Flammang
Here is some updated sample code that I'm currently using

You'll get the 403(Forbidden) status code when there is something wrong with your signature.  For me it was the dateTime and use my local time zone.

Hope this helps
//method to use the S3 webservice and save pdf file
    public void saveToS3(String binaryPDF,String docName,String bucketname,String method,String contentType,String region,String key,String secret){
        /*  Input Variable Explanations
        binaryPDF -- this is a base64 Encoded binary string representation of a PDF I get from a different web service
        docName --  any name you want to use as the saved document name
        bucketname --  the Amazon S3 bucket you are saving to
        method --  I currently only use PUT
        contentType --  I leave this blank and it seems to work fine
        region --  !!!Important, this needs to be the region that the S3 bucket is set to, for example '-us-west-2'
        key --  this is the key you generate in the AWS console under Identity & Access Management for the user account setup to access the S3 bucket
        secret --  this is the secret you generate during the same process
        */

        //setup variables
        String formattedDateString = Datetime.now().format('EEE, dd MMM yyyy HH:mm:ss z','America/Denver');    //this is needed for the PUT operation and the generation of the signature.  I use my local time zone.
        String filename;
        HttpRequest req = new HttpRequest();
        Http http = new Http();
        filename = 'TEST_BUCKET_FOLDER_1/TEST_SUBFOLDER_1/' +    //Include any folders and subfolders you are using in your S3 Bucket
            docName.replace(' ', '+') + '.pdf';   //this replaces any spaces in the desired document name with a Plus symbol '+', as the filename needs to be URL friendly
        
        req.setHeader('Content-Type', contentType);
        req.setMethod(method);
        req.setHeader('Host','s3' + region + '.amazonaws.com');  //path style
        req.setEndpoint('https://s3' + region + '.amazonaws.com' + '/'+ bucketname + '/' + filename);   //path style
        req.setHeader('Date', formattedDateString);
        req.setHeader('Authorization',createAuthHeader(method,contentType,filename,formattedDateString,bucketname,key,secret));
        
        if(binaryPDF != null && binaryPDF != ''){
            Blob pdfBlob = EncodingUtil.base64Decode(binaryPDF);
            req.setBodyAsBlob(pdfBlob);
            req.setHeader('Content-Length', string.valueOf(binaryPDF.length()));
            
            //Execute web service call
            try {
                HTTPResponse res = http.send(req);
                System.debug('MYDEBUG: ' + docName + ' RESPONSE STRING: ' + res.toString());
                System.debug('MYDEBUG: ' + docName + ' RESPONSE STATUS: '+res.getStatus());
                System.debug('MYDEBUG: ' + docName + ' STATUS_CODE:'+res.getStatusCode());
                
            } catch(System.CalloutException e) {
                system.debug('MYDEBUG: AWS Service Callout Exception on ' + docName + 'ERROR: ' + e.getMessage());
            }
        }
    }
    
    //create authorization header for Amazon S3 REST API
    public string createAuthHeader(String method,String contentType,String filename,String formattedDateString,String bucket,String key,String secret){
        string auth;
        String stringToSign = method+'\n\n'+contentType+'\n'+formattedDateString+'\n/'+bucket+'/'+filename;
        Blob mac = Crypto.generateMac('HMACSHA1', blob.valueof(stringToSign),blob.valueof(secret));
        String sig = EncodingUtil.base64Encode(mac);
        auth = 'AWS' + ' ' + key + ':' + sig;
        return auth;
    }

 
Umer Farooq 12Umer Farooq 12
Hi jason,

i have done this successfully. i want to upload multiple attachments on s3 bucket almost at one time i need to transfer 25 to 30 attachments. how i can do this ? currently i hit on my apex through scheduler and soap connection with sales force and get attachments from sf and upload it on s3. in one time it upload only 4 or 5 attachments. how i can upload multiple attachments on s3 at a time ?

Your early response will be appreciated.
Thanks 
umer
Jason FlammangJason Flammang
I don't know your specific example, but my best guess would be for you to check out Batch Apex: https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/apex_batch_interface.htm

I know there are limits as to the number of callouts that can be made duing a batch process, so you'll want to make sure your batch size keeps you within those limits.

You'll also want to make sure you specify Database.AllowCallouts when setting up your batch APEX class otherwise you'll get an error.

I currently use a batch APEX class to make callouts to a 3rd party SOAP API that only allows for one record to be retrieved at a time.  Using the Batch processing I can update thousands of records a day.  Hope this helps.
 
global class myBatchApexClass implements Database.Batchable<sObject>,Database.AllowsCallouts  {

//  run your batch code here

}

 
Kyle Dunsire 3Kyle Dunsire 3
This was massively helptful. It works a treat. Thankyou!
S3-LinkS3-Link
S3- Link is FREE App for Salesforce - Amazon Connector. Its also available on Appexchange. 
 

    Attach file related to any Salesforce object on Amazon.
    5 GB free storage for one year.
    Multiple file uplaod.
    No file size limit for upload.
    File access control capabiliy.
    Track file downloads by users.
    File exlorer capability.

https://appexchange.salesforce.com/listingDetail?listingId=a0N3000000CW1OXEA1

Here is our email address. Let us know if you have any query.
support@neiloncloud.com

Thanks.
Kenny Jacobson - Datawin ConsultingKenny Jacobson - Datawin Consulting
Anil,

I think you just have to re-arrange the stringToSign so that the "x-amz-acl" is AFTER the formattedDateString.


Yours:
String stringToSign = 'PUT\n\n'+contentType+'\nx-amz-acl:public-read-write\n'+formattedDateString+'\n/'+bucketName+'/'+filename;
Corrected:
String stringToSign = 'PUT\n\n'+contentType+'\n'+formattedDateString+'\nx-amz-acl:public-read-write\n/'+bucketName+'/'+filename;


 
Kenny Jacobson - Datawin ConsultingKenny Jacobson - Datawin Consulting
Jason, thank you for posting this!!!  It helped save me a lot of time.

And just for anyone who cares.  If you want your files to be stored in the cheaper S3 storage (Infrequent Access) and with public read-only access you would do this...

Change this
req.setHeader('Date', formattedDateString);
req.setHeader('Authorization',createAuthHeader(method,contentType,filename,formattedDateString,bucketname,key,secret));
To this:
req.setHeader('Date', formattedDateString);
req.setHeader('x-amz-acl', 'public-read');
req.setHeader('x-amz-storage-class', 'STANDARD_IA');    req.setHeader('Authorization',createAuthHeader(method,contentType,filename,formattedDateString,bucketname,key,secret));

And in the createAuthHeader method, change this:
String stringToSign = method+'\n\n'+contentType+'\n'+formattedDateString+'\n/'+bucket+'/'+filename;


to this:

String stringToSign = method+'\n\n'+contentType+'\n'+formattedDateString+'\nx-amz-acl:public-read\nx-amz-storage-class:STANDARD_IA\n/'+bucket+'/'+filename;

(Refactor as desired)
Manohar kumarManohar kumar

Hi Jason, 

Hey Jason,
Hi have to do a get request. I copied your code and made some changes. I need to get all the files from a buket.
I am getting  Status=Bad Request, StatusCode=400. Please help me with this. I am posting my code below. 
 

public void getImagesFromAmazonS3 (AWSKey__c awsKeySet, String bucketname1) {
        string secret1 = awsKeySet.Secret__c;
        string key1 = awsKeySet.Key__c;
        String formattedDateString = Datetime.now().formatGMT('EEE,   dd MMM yyyy HH:mm:ss z');
        String bucketname = 'HotelImaages';
        String method = 'GET';
        String filename = '';
        HttpRequest req = new HttpRequest();
        req.setMethod(method);
        req.setHeader('Host','HotelImaages.s3.amazonaws.com');  //us--2
        req.setEndpoint('https://s3.console.aws.amazon.com' + '/'+ bucketname + '/' + filename);
        
        req.setHeader('Date', formattedDateString);

        //get signature string
        String stringToSign = 'GET\n\n\n'+formattedDateString+'\n\n/'+bucketname+'/'+filename;
        System.debug('stringToSign::'+stringToSign);
        String signed = createSignature(stringToSign,secret1);
        String authHeader = 'AWS' + ' ' + key1 + ':' + signed;
        req.setHeader('Authorization',authHeader);
        //Blob PDF = EncodingUtil.base64Decode(binaryPdfString);
        //req.setBodyAsBlob(PDF);
        Http http = new Http();
		system.debug('req:::'+req);
        try {
            //Execute web service call
            HTTPResponse res = http.send(req);
            System.debug('RESPONSE STRING: ' + res.toString());
            System.debug('RESPONSE STATUS: '+res.getStatus());
            System.debug('STATUS_CODE: '+res.getStatusCode());
            System.debug('res:::'+res);
				
        } catch(System.CalloutException e) {
            system.debug('AWS Service Callout Exception: ' + e.getMessage());
        }
        
  }

     //create authorization header for Amazon S3 REST API
   public string createSignature(string canonicalBuffer,String secret){
        string sig;
        Blob mac = Crypto.generateMac('HMACSHA1', blob.valueof(canonicalBuffer),blob.valueof(secret));
        sig = EncodingUtil.base64Encode(mac);

        return sig;

	}
 
Pls point me in the rigth direction. I am new to this.. 

Thanks,
Manohar

 

Raji MRaji M
Hi Jason,

I have tried to uploade a file to s3 bucket using your code but I face 403 error. I tried with my local time as well but no luck. Can you please help me out in this?

Thanks,
Raji M
Prashant Menon 4Prashant Menon 4
Hi Jason,
Greetings.  This is a massive help!! to anyone attempting S3 integration first hand. I wish I had this URL when I needed to create the integration!! :)

Some key points worth highlighting:
  1. Prefer NamedCredential with AWS Signature v4.0 as one need not then expose the keys in the code. Also, authentication gets taken care from the NamedCredential (less code) 
  2. The FileName needs to be compatible to be sent over HTTPS. Prefer to remove any extra spaces (optional) and URLEncode it. This sometimes can cause issues in the signature check
@Raj - [2] may be of use for you as well.

Sample code snippet

If there is anything amiss or incorrect; please be happy to correct me.

Cheers!
Prashant Menon
 
Home AddyHome Addy
@Prashant Menon 4, can you please post complete code and steps involved to upload a file to AWS S3 bucket from salesforce,it will help the community tremendously
Daniel Best 7Daniel Best 7

Here is a PUT request using Signature Version 4 If anyone need it:

//User Variables
    String key = '{Access Key}'; 			//Your AWS key
    String secret = '{Secret Key}'; 		//Your AWS Secret key
    String method = 'PUT';					//Method you are useing, Note: Other methods may be different. 
    String bucketname = '{Bucket Name}'; 	//Your AWS bucket name.
    String region = '{Region}';				//Your AWS region name.
    String service = 's3';					//S3 is the Aws Service You are using.
    String host = bucketname+'.'+service+'.'+region+'.amazonaws.com';//Where you are posting the object.
    String filename = '{File Name}';		//File Name with no spaces! If th File Name has spaces or other unique char it will need to be Uri Encoded
    String fileType = '{File Type}';		//Type of File (i.e., application/pdf, image/jpeg, text/plain)
    Blob fileBlob = '{File contents as Blob}';	//Binary of the File.


    //Current Date and Time formatted in different ways.
    String longDate = Datetime.now().formatGMT('yyyyMMdd\'T\'HHmmss\'Z\'');
    String shortDate = Datetime.now().formatGMT('yyyyMMdd');
    String formattedDateString = Datetime.now().formatGMT('E, dd MMM yyyy HH:mm:ss z');


    //Variables User in the Signiture Process
    String algorithmName = 'HmacSHA256';
    String algorithmUsed = 'AWS4-HMAC-SHA256';
    String headersInSigniture = 'date;host;x-amz-content-sha256;x-amz-date';


    //Calculated Variables
    String url = 'https://' + bucketname + '.' + service + '.' + region + '.amazonaws.com' + '/' + filename;
    Blob paylodHashing = Crypto.generateDigest('SHA-256', fileBlob);
    String hashedPaylod = EncodingUtil.convertToHex(paylodHashing);
    String credential = key +'/'+ shortDate +'/'+ region +'/'+ service +'/aws4_request';    


    //Create Canonical Request String, Hash it, then convert to hex.
    String canonicalRequest  = 	method + '\n' + 
                    '/'+ filename + '\n' + 
                    '\n' +		//If you want to add a Path do so here, just like the filename (this will need to be Uri Encoded)
                    'date:'+ formattedDateString + '\n' +
                    'host:'+ host + '\n' +
                    'x-amz-content-sha256:'+ hashedPaylod + '\n'+
                    'x-amz-date:'+ longDate + '\n'+
                    '\n' + 
                    headersInSigniture + '\n' + 
                    hashedPaylod;

    Blob canonicalHash = Crypto.generateDigest('SHA-256', Blob.valueof(canonicalRequest));
    String canonicalHashString = EncodingUtil.convertToHex(canonicalHash);


    //Create String To Sign String, Hash it, then convert to hex.
    String stringToSign = 	algorithmUsed + '\n' + 
                  longDate + '\n' + 
                  shortDate +'/'+ region +'/'+ service +'/aws4_request' + '\n' +
                  canonicalHashString;


    //Create Signing Key by Hashing, then convert to hex.
    Blob KDate = Crypto.generateMac(algorithmName, Blob.valueof(shortDate), Blob.valueof('AWS4'+ secret));
    Blob kRegion = Crypto.generateMac(algorithmName, Blob.valueof(region), KDate);
    Blob kService = Crypto.generateMac(algorithmName, Blob.valueof(service), kRegion);
    Blob signingKey = Crypto.generateMac(algorithmName, Blob.valueof('aws4_request'), kService);


    //Create the signature by Hashing the String To Sign and Signing Key together.
    Blob blobSignature = Crypto.generateMac(algorithmName, Blob.valueof(stringToSign), signingKey);
    String signature = EncodingUtil.convertToHex(blobSignature);       


    //Create the Http Request.
    HttpRequest req = new HttpRequest();
    req.setMethod(method);
    req.setEndpoint(url);
    req.setHeader('Authorization', algorithmUsed +' Credential='+ credential +',SignedHeaders='+     headersInSigniture +',Signature='+ signature);
    req.setHeader('date', formattedDateString);
    req.setHeader('host', host);
    req.setHeader('x-amz-content-sha256', hashedPaylod);
    req.setHeader('x-amz-date', longDate);
    req.setHeader('content-type', fileType);
    //If you want to add more header Here be sure to add them to the canonicalRequest and headersInSigniture String.
    req.setBodyAsBlob(fileBlob);


    //Submit the Request.
    Http http = new Http();
    HTTPResponse res = http.send(req);
    if (res.getStatusCode() == 200) {
        system.debug(res.getBody());
    }else{
        system.debug(res.getBody());
    }
Sahana Shekar 7Sahana Shekar 7
Hi all,
I am trying to authenticate using the iam role. 
Here i dont have accesskey and secretkey. So Please post me with your code snippet as to how to use the iam role to dynamically generate the key and secret using assume role.
Also please provide the preconditions .

Thanks
sahana