Workaround Salesforce governor limits when working with files

Last updated on 
Workaround Salesforce governor limits when working with files
Custom Meta will disappear itself

get-maximum-results-from-email-marketing-with-email-trackers

limitations-of-engagement-metrics-for-email-marketing

email-marketing-is-here-to-stay-heres-why

Introduction

Salesforce is a multi tenant system. This means, Salesforce resources are shared by all Salesforce accounts. To prevent the system from abuse and to maintain good performance across all Salesforce accounts, Salesforce applies various limits on accounts & transactions. These are called governor limits.

Salesforce mentions all the governor limits here. Although there are various limits, only some of them become restrictive when working with files in apex.

In this article, we explore what are the important file-related governor limits, why they can become restrictive and finally how to bypass them using a little code in apex (or outside of it).

Who are we?

upload in progress, 0

At CloudFiles, we specialize in creating seamless integrations between Salesforce and various external file storage platforms, including Google Drive, SharePoint and others. Our mission is to simplify your data management, cut costs, and enhance customer engagement. Discover the power of CloudFiles on AppExchange and revolutionize your Salesforce document experience.

Types of Salesforce Governor Limits

Apex Limits

Apex is Salesforce's proprietary programming language used to build custom business logic and integrations. Apex governor limits regulate the execution of Apex code to prevent runaway processes that could monopolize system resources.

Key Apex governor limits include -

  1. CPU Time Limit: Specifies the maximum amount of CPU time that Apex code can consume during execution.
  2. Heap Size Limit: Governs the amount of memory allocated to store data within the Salesforce environment.
  3. DML Statements Limit: Controls the number of Data Manipulation Language (DML) operations, such as inserts, updates, and deletes, that can be performed in a single transaction.
  4. SOQL Query Limits: Restricts the number of Salesforce Object Query Language (SOQL) queries that can be executed in a single transaction.
  5. Governor limits for various callouts, email sends, and future method invocations.

Platform Limits

These limits apply to various aspects of the Salesforce platform beyond just Apex code execution. Examples include:

  1. Concurrent API Request Limits: Restricts the number of concurrent API requests that can be made to Salesforce.
  2. Email Limits: Controls the number of outbound emails that can be sent per day, per organization.
  3. Data Storage Limits: Governs the amount of data that can be stored in Salesforce databases.
  4. File Size Limits: Specifies the maximum size of files that can be uploaded to Salesforce.

Concurrent Limits

Concurrent limits restrict the number of actions or processes that can be executed simultaneously within the Salesforce environment. Examples include:

  1. Concurrent Long-Running Apex Request Limits: Restricts the number of long-running Apex requests that can execute concurrently.
  2. Concurrent Batch Job Limits: Governs the number of batch jobs that can run concurrently.
  3. Concurrent Visualforce Page Requests: Controls the number of Visualforce page requests that can be processed concurrently.

Miscellaneous Limits

These include various other limits that regulate specific functionalities or behaviors within Salesforce. Examples include:

  1. Maximum number of fields per object.
  2. Maximum number of workflow rules, validation rules, or process builder processes per object.
  3. Maximum number of custom metadata records or custom settings.

Why Are Salesforce Governor Limits Important?

Salesforce Governor Limits play a crucial role in maintaining the overall health and performance of the Salesforce platform. Here's why they are important:

Performance Optimization

By imposing limits on resource usage, Governor Limits prevent poorly optimized code or processes from consuming excessive CPU time, memory, or database resources. This ensures that the Salesforce platform remains responsive and performs optimally for all users.

System Stability

Governor Limits prevent individual transactions or processes from monopolizing system resources, which could lead to system slowdowns or outages.By enforcing these limits, Salesforce maintains system stability and prevents disruptions to service.

Fair Resource Allocation

Governor Limits promote fair resource allocation by preventing a single user or application from hogging system resources at the expense of others. This ensures equitable access to system resources for all users and applications.

Scalability

Adhering to Governor Limits is essential for building scalable Salesforce applications that can handle increasing volumes of data and users. By enforcing limits on data storage, API usage, and other resources, Salesforce encourages developers to design applications that can scale with the growth of their organizations.

File Use-cases & Governor Limits

Files are essential part of the sales process. As a Salesforce developer, you would often come across the use cases that involves working with files, mainly uploading files from Salesforce to cloud drives such as Sharepoint, Google Drive, OneDrive, Box, Dropbox, AWS S3 etc…

Here are some common examples when such file uploads might be needed when working in apex -

  1. When someone uploads a file into chatter feed or the notes and attachments section
  2. When a files comes in as an email attachment
  3. When community users upload files from their portals
  4. When sales / service teams need to upload files from a lightning web component

Salesforce has a bunch of standard objects that store binary data (blob) and are used for file management.

There are 6 main objects that are used to meet all the file management needs. These are Attachment, ContentDocument, ContentNote, Document, Folder & Note. The "Files" Object that is commonly used by sales & service cloud users uses ContentDocument object.

When you work with these objects in apex, you would generally run into governor limits like heap size & callout timeouts. Here are some common governor limits you would run into -

  1. Maximum heap size - It is the amount of memory that is used by an apex transaction. For synchronous methods it is 6 MB and for asynchronous methods it is 12 MB. This means, that if you read a file as a blob in apex, you can only process the files with size less than these limits
  2. Maximum cumulative timeout for all callouts - It is the time that all HTTP calls would take to finish in an apex transaction. This limit is 120 seconds. This means that if you’re uploading/downloading the file to/from an external service, it needs to finish within 2 minutes
  3. Maximum execution time for each Apex transaction - This limit is 10 minutes for both synchronous & asynchronous apex methods. If you’re doing long running tasks like converting a file to pdf, it needs to finish within 10 minutes

How to bypass the governor limits

One way to workaround the governor limits while uploading files to an external storage from Salesforce is using the SObject Blob Retrieve API. This is a REST API which gives binary content of any Salesforce object that has a blob field.

This method requires an external service that makes an authenticated call to the blob retrieve API to fetch the binary data and put it on external storage. Since REST API doesn’t impose a response size limit & timeouts, this successfully works around the file size limit.

Here is an example of how to upload a pdf attachment(ContentDocument in lightning) to AWS S3 using javascript.

import axios from 'axios';
import aws from 'aws-sdk';

const { data } = await axios.get(
    'https://ap5.salesforce.com/services/data/v55.0/sobjects/ContentVersion/0696D000001Pmk5QAC/VersionData',
    {
      headers: { Authorization: `Bearer ${salesforceAccessToken}` },
    }
  );

const s3 = new aws.S3({
      accessKeyId: AWS_ACCESS_KEY_ID,
      secretAccessKey: process.env.AWS_ACCESS_KEY_SECRET,
    });
    s3.upload({
      Bucket: 'UPLOAD_BUCKET_NAME',
      Key: 'test.pdf',
      Body: data,
      ContentType: 'application/pdf',
    });

One catch with this approach is that, this code needs to run outside of Salesforce using an API server or cloud functions like Salesforce functions, AWS lambda or Azure functions.

Uploading to other cloud storage providers

The same technique can be used to upload the files to Google Drive, OneDrive, Sharepoint, Box, Dropbox using the appropriate upload API and supplying the right authentication.

We recommend using OAuth2 for authentication

Thus a simple combination of a cloud function and an authentication mechanism can help you bypass governor limits and transfer any types of files from Salesforce storage.

There are 3rd party tools that make this extremely easy.

Upload using a 3rd party tool

We at CloudFiles have built an AppExchange app to work with files easily in Salesforce. Using CloudFiles, you can avoid setting up an API server, setting up authentications or building integrations with different cloud storage platforms.

The same functionality described above can be achieved using CloudFiles with one line of Apex code -

cldfs.Client.upload(
      files, // a list of sobjects with blob field
      'rTmK2YQnxaEIFkU_C18a62b2xWyDB', // folder id in external storage
      'google', // destination: google|onedrive|sharepoint|box|dropbox|cloudfiles(s3)
      'mydrive', // drive id in external storage(optional)
      null // site id in case of sharepoint(optional for other storages)
    );

Users can authenticate Salesforce & destination file storage using the CloudFiles UI. It’s one time, one click setup process.

You can use the upload code as part of flows or apex triggers to automate the file management in Salesforce. All this without ever hitting the governor limits!

Happy coding!