Karishma Lalwani is a Senior Director of Project Management here at Salesforce. Today I’m talking with her all about scalable architecture. In our conversation, we discuss what scalable architecture means on an enterprise level and some takeaways for developers.

Karishma was also once known as the “queen of encryption,” a title her former coworkers gave her as a result of her work on a pilot product for platform encryption. Karishma has a long history helping customers be successful at scale and advocating for them internally, tune in to hear from her wealth of knowledge and experience.

Show Highlights:

  • What the frontier scale team does.
  • What elastic infrastructure is and how it helps meet demand.
  • What connected platforms are.
  • How the “always on” model works.
  • Best practices for developers working with SOQL.
  • What not to do as a developer working with a 10 record query and why.
  • Why row locking and sharing are problematic.
  • What developers need to know about “bulkifying” code.
  • What big objects are and how developers can use them to scale.
  • What Vaccine Cloud is and the challenges it presented to architecture.
  • All about Karishma’s upcoming podcast.

Links:

Episode Transcript

Karishma Lalwani:
Very, very unique back then, Cloud computing. We’re able to talk about technical architecture. Since it was CRM-based as well, and Siebel was all CRM, it made complete logical sense.

Josh Birk:
That is Karishma Lalwani, a senior director of product management here at Salesforce. I’m Josh Birk, your host with the Salesforce Developer Podcast. Here on the podcast, you’ll hear stories and insights from developers, for developers. Today, we sit down with Karishma and talks about scalable architecture.

Josh Birk:
What does that mean on an enterprise level and what are some takeaways for developers? We start, as we often do with her early years here at Salesforce, and how she got a certain title.

Karishma Lalwani:
As an architect. I pretty much got all my certifications and ramped up on the technology within the first six months, I would say. Back then, before being billable on a customer, obviously you need to know the platform. We do a pretty good job of getting the architects trained and surely got into a lot of projects very quickly.

Karishma Lalwani:
I would say two years ago, I transitioned from being an architect to being a PM on the other side of Salesforce. It’s been an interesting eight-year journey so far.

Josh Birk:
Now, before we talked about your current job, tell me why your former colleagues call you the queen of encryption?

Karishma Lalwani:
I hope they still remember me. The thing was, this was again in 2014, we were rolling out this new pilot product for platform encryption. I spent a lot of time back then with the product team to make sure that the product is enterprise-ready. I used to be deployed in these large complex enterprise customers, their needs were super critical, super important.

Karishma Lalwani:
If we were able to prove that the product works for one big financial services customer, surely we can have a path from pilot to GA. I almost spent two years, I would say, with [inaudible 00:02:30] who was the PM back then, and almost like co-creating getting feedback, real feedback from my customer, and getting the product enterprise-ready.

Karishma Lalwani:
As part of that, then when you start getting that deep into some technology, you automatically start answering a lot of community questions. The word spreads around in terms of, “This person is the subject matter expert in this area.” That’s how the whole queen of encryption data came to be.

Josh Birk:
I love it. Let’s talk about that a little bit, because it sounds like you’ve had a lot of experience with architecture, and just big scale thing. You said you just got into project management, but obviously you’re still very centered on architecture. Tell me about your current role in about frontier scale team.

Karishma Lalwani:
Frontier scale is a very unique team in our product organization. Essentially, we are a team that works with these largest enterprise customers. Some of them who are doing things at a massive scale that we have never seen before.

Karishma Lalwani:
The intention of the team was, “Let’s see who are these customers who are really, really stretching our platform and breaking all barriers when it comes to performance and scale and let’s see, what would it take to make them successful?” As part of my job, it’s a dual motivation. One is customer success.

Karishma Lalwani:
Make sure that the customer actually can deploy very complex business processes on our platform. The second part is product success, which is, bring back these use cases from these real enterprise customers back to the product team. Be that voice of the customer. Really drive enhancements when it comes to what would it take for a product to scale at that level.

Josh Birk:
There’s a little bit of outreach and trying to keep the customer safe, but then there’s also when our bad, you come back to Salesforce and try to get that fixed?

Karishma Lalwani:
Exactly. We are working with these one-off Snowflake use cases, you learn a lot. You’re probably the first one doing something, you get some patterns, you get anti-patterns. You get product enhancement requests, that in turn drives our product to a much more large scale enterprise readiness as well.

Karishma Lalwani:
I work with almost, I would say, 10 to 12 PMs within our product organization. Really getting deeper into the technology stack in core. Understanding where that ceiling is and what can we do to raise that ceiling even further, so that these customers are ready.

Josh Birk:
Well, let’s talk a little bit more about that specifically about kind of the core and the architecture around the core. You’ve described elastic infrastructure connected platforms and always-on experience as something vital to the success of that. Let’s walk through a little bit. Starting the first one, what is elastic infrastructure? How does that help meet demand?

Karishma Lalwani:
When it comes to elastic infrastructure, our enterprise customers have ever-growing needs. Sometimes, we can predict what situations they would end up in. Customers who are doing global expansions, so they’re not just one country, but multi-country implementations. They’re launching new products.

Karishma Lalwani:
It could be a merger and acquisition scenario, where there is a big company trying to merge with another big company. What scale challenge would that intake? When it comes to these growing need for this auto scale, adding on-demand capacity, this is something that is a very, very common use case that we’ve started seeing. It’s also to do with seasonality of it.

Karishma Lalwani:
Let’s say, we have a customer who is already a customer, and they have events like Black Friday. You have Intuit, who’s doing the tax season. Those customers have elastic infrastructure needs, only for a shorter time period. How do you architect for those key seasonality events. Even generically speaking, when you’re trying to look at a merger situation, what would it take for you to scale?

Karishma Lalwani:
Those are some of the use cases that I have spent a lot of time researching, investing, and coming up with the right strategies of how customers should think about it early on, so that they don’t have to face any scalability challenges once they go into production.

Josh Birk:
Got it. What are the working parts of a connected?

Karishma Lalwani:
Connected platform, a couple of things, again, when it comes to these enterprise customers, you’re not dealing with one system, you’re dealing with multiple systems. As well as you’re dealing with multiple ways the customers use their system. It could be today, the world is so connected.

Karishma Lalwani:
We have different devices. We have different ways to reach a customer. If I were to put myself as a customer with having, let’s say, a cell phone. I’m facing some issue, or even let me start early on, but I’m just trying to figure out what network should I go with? I possibly start at a website where I’m just browsing different competitor websites.

Karishma Lalwani:
I’m trying to see who is giving me a better rate. That in itself could generate a lead for that specific company to say, “There’s this person who’s trying to look at your pricing and packaging. Why don’t you do an outreach?” Sure enough, nowadays, you will find that even if they get certain pieces of information from you, you will be contacted.

Karishma Lalwani:
You will be asked what your needs are. Then, it means that your profile is now part of that system. You started the website. You get a call from the company. You make a sale, let’s say, over the phone. Then, you get an email saying, “Hello, customer. Welcome to so-and-so network.”

Karishma Lalwani:
That is what I mean by connected platforms. Behind the scenes, there are these huge engines. One is a completely different marketing and advertising platform, which is connected to customers sales technologies, which could be a Sales Cloud implementation, which is connected to then service in case I have an issue. How do all of these talk together?

Josh Birk:
The similarities I’m hearing is freaking me out a little bit because it’s like, I’m hearing the interview ahead with Andrew Lawrence, where he was talking about how do you survive the holiday season on commerce Cloud. Then I think his phrase was something along the line of the new storefront is your phone.

Josh Birk:
People aren’t even going necessarily to their desktop browsers to purchase things anymore. They’re just doing it in their hands. I think those trends are really interesting. I also have a call back to Abraham David Lloyd talking about the digital persona for a B2C architecture.

Josh Birk:
How do you set all that kind of stuff? I love all of that. Tell me a little bit more about being always-on, when it comes to an enterprise platform.

Karishma Lalwani:
Always-on is something to do with mission-critical apps. Let’s say there’s a 24/7 call center around. We want to be able to predict any outages in advance and prevent that. Our customer demands are definitely shifting to this always-on model, which is basically deploying these business critical and mission-critical apps, when it comes to customer service.

Karishma Lalwani:
Let’s say, if there is like an insurance company, whose built their business model on Salesforce. Let’s say there is a hurricane that comes in south Florida. Suddenly you have a need for people, and there’s an influx of claims that is coming on to Salesforce. What if I’m there and I’m looking at my house and I need help.

Karishma Lalwani:
I need my company to be able to help me. That is the mindset of having an always-on infrastructure. Imagine if at that point you were going to this app and the said, “Sorry, we are on a six-hour maintenance window. Please file a claim on 12:00 PM,” something like that. What the customer experience looks like in that case?

Karishma Lalwani:
Same thing with government and federal customers. Many times we have, let’s say, the situation right now. Suddenly, we are dealing with vaccinations and there are state laws that change every hour. What if California decides to increase the eligibility? Now it’s always-on people just want to go and get their appointments.

Karishma Lalwani:
How do you manage that sudden surge in the system, that you anticipate sudden scale? How do you make sure that your system is able to support that experience?

Josh Birk:
There is some of the broad challenges, especially for us, as a Cloud company. Let’s flip the perspective around a little bit and talk about the little guy or person, not to gentrify. The developer in the chair who is going to be working on some of the stuff.

Josh Birk:
I’m a developer I’m dealing with a large, complicated data structure. What should I be thinking about in order to keep that SOQL performance? What are some best practices there?

Karishma Lalwani:
Absolutely. If we’re talking at a very, very beginning of the implementation cycle, what are some of the things that you need to take care? If I’m a developer, having the performance and scale mindset early on is very critical. Oftentimes, what happens is we would write a SOQL query example to meet the needs of let’s say, 100 records.

Karishma Lalwani:
We don’t know what that same query would do if you inject one billion records. If I’m a developer, my sandbox that I play with is pretty small. I don’t have the ability to even generate one billion records and even test at one billion records. How do I know that it’s never going to work?

Karishma Lalwani:
Also, when you are in that mode, you’re often your priorities, “I need to deliver this and I need to deliver this fast because I need to meet a certain deadline.” Then, how do we think about scale early on is very critical. To those people, I would say there are tools that are available today. You have a way to find out what your query path is going to be.

Karishma Lalwani:
You have a way to find out what the cost of the query is going to be. There’s very little importance at that point when you’re doing a unit test that you would pay attention to, but those things are extremely critical when you think about what would it do to the scalability of your application?

Josh Birk:
I’ll keep it short because I know I’ve given this cautionary tale on the podcast before. I think back to a customer who shall not be named, who we started designing, knowing it was going to, because there was three databases involved, or something like that. We knew even from a Salesforce point of view, that once we got to that data, there was going to be a lot of it.

Josh Birk:
At some point during the development, I’m like, “Please give me something that is like your production data.” They gave me something with 500 rows. I went back to them like, “There’s no way we’re dealing with, give me something that looks like your production data.” They gave me something to add 5,000 rows. I’m like, “Okay, whatever.”

Josh Birk:
Then, we try to launch and the actual row account was more half a million. Performance was suffered a little bit. Actually, let me ask you a question I asked Neil, putting it into data terms. When you say big, when you say, what is a big data set in your head?

Karishma Lalwani:
It’s very subjective. There are different ways people look at it. For me, it’s not how much of data do you have resident in Salesforce. You can have up to a half a billion records in Salesforce, in your accounts, in your contacts. That’s all okay. It’s what you do with that data, how you use it that becomes more critical.

Karishma Lalwani:
Just 500 million records in an account object sitting there doing nothing does no damage to anybody whatsoever. Let’s say now you build a SOQL that’s trying to get all the account records with first name John. That is a big problem. That’s why I said it’s subjective. It’s not to do with how much data is big. Even with 10 records, if you have a crappy query, guess what? It’s not going to scale.

Josh Birk:
Let me actually ask you on that a little bit. What’s something, maybe a newbie developer’s going to do wrong on a SOQL that could make a temporary records, a query, not be as fast as it should be?

Karishma Lalwani:
Not having any filter criteria as part of your work class.

Josh Birk:
For those people who aren’t familiar necessarily with the niceties of database design, why is that?

Karishma Lalwani:
That is because that would end up doing a full table scan, and you don’t know what the dependencies are. How your relationships are mapped? If you’re trying to query a child record and you have a dependent field, then that will do a multiple level, adds the multiple complexity into that relationship as well.

Karishma Lalwani:
When it comes to data profile, it’s very important to know, not just what you’re doing for that specific object, but how other relationships built around that as well. Selectivity of the query is extremely important, not just selectivity, pure database concepts anywhere, not just Salesforce. Make sure you’re using index fields. We all learn that.

Karishma Lalwani:
Make sure you are able to find the rugged as fast as you can and as correctly as you can. The number of criteria is that you add, number of conditions. Make it as specific as possible would be the first goal.

Josh Birk:
Now, I think a lot of developers and I’m going to include myself in this because I have definitely found guilty of the play around a bunch of the developer edition. Then, try to move something into a Sandbox production, and realize that there are strange scenarios that come up.

Josh Birk:
I have run into this, but I don’t know if a lot of developers consider things like row locking and sharing. How do those come into play? When can they become problematic?

Karishma Lalwani:
That’s a very good question. I would tell you, 90% of the customers that we work with, where they say, “Oh my God, our data is not loading or it’s loading very slow.” Row lock is the number one problem.

Josh Birk:
Yeah. Doesn’t surprise me.

Karishma Lalwani:
That goes back to your data profile. What are you trying to do with the data? If it comes through row locks, obviously the very first thing that we recommend is if you’re trying to do a massive data load, just sought the records by the parent ID. If there are different records trying to lock the same parent, guess what? It’s going to take time.

Karishma Lalwani:
Simple techniques that we have to maximize the paternalism and the load timings that customers can take care of, so that the parent child is not being the problem for the row lock. Again, coming to sequencing of the data, what are your child entities? How are they referencing the ownership ID, or the parent ID?

Karishma Lalwani:
Which records do you need to load first, which is referenced data versus which is not? Those kinds of things that are very easy to think about and to actually get it in place before you even approach loading of data in Salesforce. Then, what was the second thing? I’m trying to recall. You said row lock and one more?

Josh Birk:
Sharing.

Karishma Lalwani:
Sharing. That’s a second favorite. This is something that’s extremely unique to a platform. Me personally, when I started on Salesforce, this is one of the topics that I had to really, really understand a lot more. Anything that you do when it comes to sharing, I’ve worked with big banks where as part of their business use case, they have quarterly account adjustments, which leads to massive role hierarchy changes.

Karishma Lalwani:
It could be as simple as your sales organization undergoes a restructuring every quarter. You do different territory mappings, or even as simple as your employees or your bankers change from one branch to another. Oftentimes, we see that customers map all of that as a role hierarchy set up in Salesforce.

Karishma Lalwani:
Now, when it comes to massive recalculations like that, guess what? If you do it all at once, and if you have not tested it, chances are users are not going to use your system all weekend. I have been in those calls. In fact, funny story, I had once an architect leader in the customer side. He said, “You run that means it’s going to take 16 hours.”

Karishma Lalwani:
I said, “No, what I told you was, please test it in a full copy Sandbox.” Simple things like these, always test. Don’t make production your testing ground. Sharing is so unique. Simple strategies for that would be just deferring sharing calculations where it makes sense. Again, you do your adjustments first and the sharing recalculation happens later, asynchronously.

Josh Birk:
Just to clarify, when you say all weekend, you were not exaggerating there.

Karishma Lalwani:
I was not, if there is one thing you should test is it has to be any massive sharing changes that you make to your object model.

Josh Birk:
I’m trying to think of how to frame this question. What does this look like from an APEX point of view? I think every developer has probably heard like bulk applied by code, but what have you seen that’s maybe I, as a developer thought I bulk applied by code, but I didn’t think of XYZ. What would you tell developers to know about [inaudible 00:23:17], I’m going to try another verb.

Karishma Lalwani:
It starts with even the trigger structure. It’s not just bulkification. Yes, bulk applied by code is the number one thing that we asked to do. It’s just to make it more efficient. Let’s say you have designed a business process that is doing something on the opportunity object and you have bulkified your opportunity trigger doing sold records.

Karishma Lalwani:
You’d think as a developer, you’re the smartest developer and good job. Then, there is another admin who’s created a workflow on opportunity, and that does some updates too. There is a flow that is built by a new person on the team who’s again, using opportunity as the object. Now, what happens to [inaudible 00:24:12]?

Karishma Lalwani:
You might have written the best code for a given situation, but have you taken a step back and look broader to see what are the other process automation technologies that you have that Salesforce provides, that someone else would have used? There was this customer that, I would say 2,500 lines of code on one object.

Karishma Lalwani:
Given the requirement, the developer was very quick to solve it and write four more lines, add five more lines. It took a simple review to show the person, what you’re trying to do is already done on line 75, so you might want to just use that variable.

Josh Birk:
When you said that number of codes, we’re on a podcast so people can’t see how much I actually shivered.

Karishma Lalwani:
How do you manage the technical data? You have been building for years and years, but are you really doing housekeeping, simple housekeeping items that lead to APEX challenges.

Josh Birk:
The lean is to always be additive. Over time, that’s going to just potentially cause more problems.

Karishma Lalwani:
Absolutely. Then, we have some other great technologies when it comes to just from a skill standpoint, not going to the database for everything. Reuse what do you have already queried in your code. Reuse whatever you’re extracting, or even use Platform Cache. It’s there, it’s a product.

Josh Birk:
I think I’m paraphrasing Neil here when he said something like the best database query you could write is the ones you don’t have to write at all.

Karishma Lalwani:
Exactly.

Josh Birk:
Well, let’s talk about one of those similar products. I’m going to confess. I have never used Big Objects, what are they and how can developers use them to scale?

Karishma Lalwani:
Absolutely. I would tell you what use cases I have seen people use it successfully. Think of data that you’re not really using on a daily basis, but you still have some compliance or regulatory requirements that the data needs to reside in Salesforce. Things like closed investigations or cases, or accounts that have been terminated, but you need to have a view into them.

Karishma Lalwani:
Records that probably are better suited to have a unified view. I want to see in the last five years, how many cases have been raised for a customer. You don’t have to necessarily have all of those records in active storage. Your compliance group might come back and say, “I just need to see two years worth of data, but I still need you to hold five years for any queries or inquiries that come from a legal standpoint.”

Karishma Lalwani:
Then that kind of a situation where you do archiving strategy or data tiering, where you have some of it in active storage, in your regular customer and standard objects. Then, you define a criteria where you push everything else to Big Objects. At the same time, you have everything in Salesforce, so guess what? You can build one view to view all of it together.

Josh Birk:
See, I love all of that because I feel like it comes full circle, because I assume by pulling them into that, you’re also pulling it out of the lake of data. It can’t be a target of my bad SOQL query to begin with.

Karishma Lalwani:
Absolutely. Also, you’re not doing a lot of data sinking. The number one problem, not problem, use case is people want to bring data into Salesforce all the time. Then, they want to delete data from Salesforce all the time. I’m like, “What is your strategy? What are you trying to do? Why bring it if you want to put it back in your backend? Why don’t you just keep it there and use something like a Salesforce Connect to view it?”

Josh Birk:
Well, that’s how I feel like that deserves a shout out to external objects, which can set aside the Big Object strategy.

Karishma Lalwani:
Absolutely. I can’t tell you how many number of times I’ve had these conversations where customers are like, “We need to bring everything.” Why? “Oh, no, because we don’t want to go back and forth.” You’re bringing it, but are you going to use it?

Josh Birk:
Exactly. I think my most common question to all of my, well, the vast majority of people I consulted with was, do you really need that? I know you think you need that, but do you really need that. Can I talk you into a better car kind of thing?

Josh Birk:
Let’s talk about some real world examples. I know there’s some specifics we probably can’t get into, but tell me a little bit about the Vaccine Cloud and how it presented some unique challenges when it came to architecture?

Karishma Lalwani:
Absolutely. That is one of my favorite case studies, because your solution where you’re trying to do something to millions of people. You know that it’s to be a situation where you are going to face maximum concurrency. Everybody wants it and wants it now, and you can’t deny service to anybody.

Karishma Lalwani:
How do you architect that? Some of the principles for that specific piece, again, it comes to a lot to do with how do you cash entities? How do you manage the flow where you’re bringing in people from an experience standpoint, you’re bringing them in. You’re trying to get some basic information upfront, which needs to be more synchronous.

Karishma Lalwani:
Then, you have a lot of other things that can be done asynchronous, that way you’re reducing the shortened synchronous activity time, which is holding the database. How do you think about architecture patterns, where you’re almost like taking the bare minimum, what are the five things I need to know from you to give you an appointment?

Karishma Lalwani:
Now that I know, how do I get an appointment? Now, that you have an appointment? What are the things that I need to do after you have booked it? Sending you an email confirmation or text confirmation. How do you decouple those transactions?

Karishma Lalwani:
Not from a experience standpoint, from a user standpoint, it still looks one single flow, but from an architecture principle standpoint, how do you do it in a way that you don’t end up choking the system?

Josh Birk:
What’s your favorite lesson learned from it?

Karishma Lalwani:
Test.

Josh Birk:
I’m sensing a theme by the way.

Karishma Lalwani:
Yes. Scale test is my favorite word. Not because I’m part of the scale team, but I’ve seen customers who, yes, followed the test, when production [crosstalk 00:31:41]

Josh Birk:
Boom.

Karishma Lalwani:
Did okay, I would say. Let’s say two weeks later, they wanted to add a new enhancement, conveniently bypass testing, just because you told me to do it the first time, not the second time, and that’s led to an outage.

Josh Birk:
Got it. Design for scale, developer scale, test for scale. Maybe not even in that order, but always do those things, at least.

Karishma Lalwani:
Absolutely. Performance and scale, that should not be an afterthought. It needs to be handled as part of your design principle.

Josh Birk:
Now, you also having a project with the wonderful Kelly Walker, a podcast named Designing Architecture on the Customer 360 Platform for Salesforce architects. Tell me a little bit more about that.

Karishma Lalwani:
We are coming up six mini episode podcast where we are going to go into different topics. All ranging around, what would it take for customer architects to design and build on Salesforce platform? We would be talking about things like integration strategies.

Karishma Lalwani:
Including things like platform events, change data capture. Anything that comes to mind from an integration standpoint, we’ll have a full episode dedicated to that. We will also be talking about topics like security. What does data security mean to you? What do we do in terms of auditing?

Karishma Lalwani:
How do you protect the data in Cloud and also some exciting new stuff, Hyperforce. Getting into the technical architecture discussions of what is Hyperforce? What are the use cases? What are the strategies? How do you go about designing and building on a Hyperforce?

Karishma Lalwani:
It’s going to be a lot of different things, I would say. I’m going to talk to the product managers, who owned that area to hear it straight from them, and just have a little conversation about what they think, and we should be doing. Also, we are going to source a lot of the questions from the community.

Karishma Lalwani:
Any architects out there, look out for some posts coming up where we will be asking you, what is it that you would like to hear on these topics?

Josh Birk:
That’s our show, that podcast Karishma is talking about is coming soon scheduled to come out in August. Keep your ears open for that and keep your eyes out on our social for more information. Now, I did ask after Karishma’s favorite non-technical hobby.

Karishma Lalwani:
Non-technical hobby? I discovered it just during the lockdown. I think a lot of us probably found new hobbies.

Josh Birk:
That has come up a lot on the show. It’s true.

Karishma Lalwani:
I got into writing. I didn’t know how much I like journaling. I used to think I’m just good at putting my thoughts together, but then I started writing some poetry. Then, I started rhyming and then I was like, “Oh, this is something that I can belong to.” Whenever I have some free time, I just go back to my little scratch pad and start writing.

Josh Birk:
I want to thank Karishma for the great conversation and information. As always, I want to thank you for listening. Now, if you want to learn more about the show, head on over developer.salesforce.com/podcast, where you can hear old episodes. See the show notes and have links to your favorite podcast service. Thanks again, everybody. I’ll talk to you next week.

 Get notified of new episodes with the new Salesforce Developers Slack app.