Ian Varley:

So the work that I do ends up, I think, being a lot more influenced by my philosophy pedigree than my computer science pedigree. And I think there are lots of people who understand how to make really good algorithms and build things really well. And there are not as many people who can sort of take that step back and say, well, why are we building this? And what is this for? And is this the right approach? And is this the right metaphor, and the right mental model, and all that stuff?

Josh Birk:

That is Ian Varley, a principal architect here at Salesforce. I’m Josh Birk, a developer evangelist with Salesforce. And here on the Salesforce Developer Podcast, you’ll hear stories and insights from developers, for developers. Today, we sit down and talk with Ian about Salesforce’s multitenant architecture, something he has been working on for the last 10 years, and comprises the core of what Salesforce actually is. And to kick things off, we asked the question, well, what is multitenancy?

Ian Varley:

A really good question. An article out there that I published called L33T M10y. And I’ve been trying for years to get everyone to use a, what’s the word for when you have an abbreviation that puts the number of letters in the middle and a numeric?

Josh Birk:

Oh, I don’t know.

Ian Varley:

There’s a word for that.

Josh Birk:

Like LE4ET or something like that.

Ian Varley:

But more like internationalization, right. Where you-

Josh Birk:

Oh, got it. Right. Right.

Ian Varley:

And so I’ve been trying to get people to use M10y for multitenancy because it’s 10 letters more. And literally no one does, it’s just me. But I’m going to keep working on it. So the basic idea of multitenancy is all software that you use is multi-user software, right? Meaning if you think about something like Gmail, they’re not running a copy of Gmail for every single person that has a Gmail account, right? They’re running one big instance and it has all these different users in it.

Ian Varley:

Multitenancy is one step beyond that. Instead of just individual users who use the system as if it’s isolated, you have groups of users who use the system as if it’s isolated, right? And so this could be for example, everyone in a company. And so, if you’re a company that buys Salesforce, then you are a tenant and you have a bunch of users and a bunch of configuration and a bunch of data that all of those users share. And the fact that it’s a multitenant system means we have these tenants and we have lots of those tenants being run on the same hardware and software concurrently. So that’s the core idea of what multi-tenancy is.

Josh Birk:

So side note, I think it’s kind of funny that the most common phrase for that tenant is, people talk about it in terms of a Salesforce org. Which sounds it’s very jargony, and it confuses people when they first hear it, but it’s actually a very on the nose name for what you’re talking about, right?

Ian Varley:

Yeah. Organization. And it’s funny because, across the different parts of Salesforce that have come in through acquisition, there are different names for that thing. Right? And so some parts of Salesforce, call it a member, or an enterprise, or a site, or a realm. There’s all these different words for the same concept, which we’ve worked towards kind of aligning those over the years. But yeah, the CRM core kind of terminology for that is an org, or an organization.

Josh Birk:

Right. And of course, I just have to give a shout out to the World of Warcraft players out there. Okay. So architecturally speaking, how are we actually defining that tenant? What kind of hardware, what kind of virtual machines are we talking about?

Ian Varley:

Well, so the tenant itself, because this is all sort of software based multitenancy, the tenant isn’t actually represented in any actual instantiation of hardware, either physically or virtually. So there’s not for example, a set of Docker containers somewhere that is your org.

Josh Birk:

Got it.

Ian Varley:

The way that it actually works in the CRM product is there’s a table called organization, and there’s a row in that table for your organization. And it has an ID. And then every other table in the system has a column called organization ID that has that value in it if it’s your row. And every single query in the system adds where organization ID equals x. At some level, it’s as simple as that, right, that you have all of these tables and so forth and the system. And there’s other things we can get into about the meta databaseness of Salesforce, which is interesting to talk about. We can talk about that in a minute. But just from the multitenancy perspective, it’s really kind of as simple as that.

Ian Varley:

And that means if there’s a container somewhere running an application server, for example, each individual requests that comes into that application server is going to have some context, right? It’s going to have some authentication of who’s the request coming from, and is it authenticated? And then as part of that, they’re going to say, okay, the organization ID is x. And that gets carried along with that request as it travels through the call stack of the application server. And so you could say, each function it enters, there’s a context object that goes along with it. And that context object, you could say, who’s the user, and who’s the organization? And those get automatically bound to any kind of database queries that are going on. And so it’s pretty much impossible for that request to access anything other than the resources to which it is authenticated and authorized.

Josh Birk:

Got it. And if we’re flipping that around, how do we go about grouping those kinds of services? When we think about it as a group of things that the user can use on the system, what does that start to look like?

Ian Varley:

Yeah, that’s a great question. And of course there are a pretty large number of individual software services that make up Salesforce. For a lot of what most people interact with, which is to say Sales Cloud, Service Cloud, Communities, the Salesforce platform, there is really one pair of dominant services. That is the Salesforce application server and the database.

Ian Varley:

And if you really had to break it down to what happens to a user request. Yes, there are lots of other systems and services that it passes through, like load balancers and caches and so forth. But the heart of it is really these two. And the request comes in, it lands on an application server. The application server says, okay, I’ve got you, I know your context. I know what database to connect you to. And I’m going to connect to the database run queries as defined by the code path, or whatever your request is, and return those results to you.

Ian Varley:

Now in very broad strokes, that’s most of what happens for these core requests. But then, when you think about how that can spiral out into lots and lots of other services, I’ll just take one example. Think about the way that search works. So when you log into Salesforce, and you say, I want to search for banana. There’s a couple of ways you could implement that. Because presumably you want to find banana in whatever database tables it is. So you could theoretically say, oh, okay, user wants to know about banana, and then iterate through every single table in the system, and say, run a SQL statement that says select all the records where name contains banana, or address contains banana, or this field contains banana, right, and so forth.

Ian Varley:

Now I’m being facetious because obviously that would never work. That would be completely impractical. So the way that search works is there’s a background process that every time data’s changed in any way, it indexes that data. Meaning it takes all of the values from the record that was changed, and it creates a reverse index of that, where it says, okay, if the answer to your name is John Banana, then I’m going to take this last name field that says banana, and I’m going to put an entry under banana in this index. It says, if someone looks for banana, there’s a record here that has banana in it.

Ian Varley:

Now that indexing service is a separate service, right? That’s not the application server, and it’s not the database. It’s the whole separate service running on hardware somewhere that gets this stream of data and does this. And then, just picture that also when you actually do the query, do the search query for banana. Again, it’s going to spawn that out to another service that knows how to look at those search indexes and return the results with the pointers to the records that you want to be looking for. So that’s just one example among hundreds of sort of specialized services that are part of this overall a suite of things that power the software that you use.

Ian Varley:

And of course, because it’s software as a service, and I guess this is something I should have said before, fundamentally, you don’t care. As the person using the software, you don’t know nor care how many services it’s broken up into, and how that stuff is all partitioned. And those are engineering problems and optimization problems for our team. But from the outside, it should look very much like, well, I just do stuff and it just works. It’s magic.

Josh Birk:

Yeah. Because going back from multitenancy before, we are a multitenant platform as a service. And so the user’s really only supposed to be concerned with login.salesforce.com and we handle the rest.

Ian Varley:

Yeah. And then all of the things that we have to deal with, like getting on the phone with a database vendor. Just last week, one of our networking subsystems, and I won’t mention which one, had a P zero, a zero day security vulnerability that just came out of nowhere. And it affected hundreds of physical, software appliances in our data centers. And there are a whole lot of Salesforce customers who have no reason to know about that and never will know about that because we just handled it. Right?

Ian Varley:

We just said, okay, we’ve got our team. We’re going to go out there. We’re going to patch these things. We’re going to test them. We’re going to make sure everything works. And then it goes back up and running. And there’s no downtime. There’s no interruption for people. But there’s a whole lot of burden there. And this is the whole point of software as a service, isn’t it, right? You’re essentially, as a service, I am selling you not worrying about all that crap.

Josh Birk:

Right. You’re the consumer. And how we built it, boxed it, got it to you, you don’t care. It’s like an Amazon box. All you really care about is that it showed up at your door.

Ian Varley:

Yep. So that’s the power of running a software as a service. Now, the power of running a multitenant service is a little bit different than that in the sense that, again, customers don’t care whether the hardware that’s running them has just them, or them and 10,000 other customers. However, they don’t care until they do.

Josh Birk:

Right. You don’t care about your neighbors until they get really loud and rowdy.

Ian Varley:

Exactly. And we do, from time to time, have issues where, because Salesforce is a platform that’s so open-ended, you could do so much with it. You can write code, of course, as people listening to this podcast know, that runs on Salesforce servers and does pretty non-trivial things. But think about that from a resource perspective. That is really physically a CPU somewhere. And that CPU is shared by hundreds or maybe thousands of tenants at the same time.

Ian Varley:

And so if that runs away with a bunch of resources, well, do all of your neighbors suddenly, are things running sluggish for them, right? So a big part of our responsibility as the service provider in a multitenant service is to make sure that that doesn’t happen, and that usage is kept within resource limits. And that we’re monitoring to say, oh gosh, this one customer is heating up here. There’s lots of examples I could cite. But let’s say you provide services to people who want to file their taxes, for example, or services to people who want to go shopping on Black Friday. Right? There’s a big spike in usage from one tenant on those days. And it’s our job to make sure that that doesn’t reflect on the usage of everybody else who’s using the system at the same time. And we have to play lots of tricks to do that.

Josh Birk:

Yeah. So let’s talk a little bit about the structure of all of that. And first, I saw this term in one of your presentations, and it kind of just tickles me a little bit. I think Steven Tamm coined the term mesoservices. What exactly do we mean by that?

Ian Varley:

Yes. Mr. Tamm did coin that phrase. The hot term in the industry these days is microservices, right? And actually that’s hot term from a few years ago. The hot term now is serverless, or whatever’s after that. But microservices, of course being a pretty, pretty trendy thing to do where you say… I mean, like I mentioned before, Salesforce does run plenty of what you would properly describe as microservices, like an indexing service for search indexing, right? That is exactly a microservice.

Ian Varley:

And the trend is okay, well, rather than having big monolithic pieces of software, we’re going to have these smaller purpose-built independent individual services, and they all interact. And that’s all well and good. And it carries its own burdens in terms of managing a huge number of services and letting them interact and debugging problems, and other things like that. But it’s a perfectly viable architectural pattern.

Ian Varley:

However, when you think about what Salesforce is and does, I mentioned that core application server, one of the really cool things about the kind of core suite of products from Salesforce, including sales and service and so forth is because of the platform nature of Salesforce, we open it up for customers to do pretty interesting and novel things on the platform as far as kind of piecing together and combining functionality. So you could do things like save an object, and run some validations on it, and then post to a chatter feed, and all these other things, kind of all in one piece of code, one piece of Apex code, you can do that for example.

Ian Varley:

And that’s really cool, but what that means is that all of those participating pieces that do all of those functions need to be callable code within the same executable. And because of the database transactional nature, there’s ways you could do that without invoking a single database transaction, but they tend to be a bit complex and brittle. And so the way that this is implemented for Salesforce is a monolithic service. It is a fairly large kind of all encompassing service that does a lot of this, which is the application server that I mentioned before.

Ian Varley:

And so when we think about there are certain microservice patterns that apply very much to Salesforce, but we have to think about it in kind of the more nuanced and complicated world of interactions of microservices and mesoservices, or macroservices, or whatever you want to call them. There’s different challenges associated with both, but it’s always going to be a mix for us just because of the nature of the application that we run.

Josh Birk:

I feel like somehow we’ve created monolithic plus plus or something.

Ian Varley:

Yeah. We talk about that a lot because everybody likes to kind of rag on monolith. And they have their problems, for sure. Building the Salesforce application’s not trivial and working in it is an acquired taste. But that said, the things that it lets our customers do is really impressive. And the things that it lets us… It’s sort of this multiplicative value because of all the things. I mean, I haven’t even mentioned beyond the platform, there’s also the entire ecosystem, right, of ISVs and so forth. And so to be able to kind of support and engender all of that, that’s really where I guess the kind of cohesive nature of the platform really matters. And so, for us, the fact that this is a monolith actually has a lot of advantages that would be extremely difficult to replicate if we tried to break it up into a huge number of smaller services.

Ian Varley:

And we’re already seeing some of the challenges as we try to integrate and include parts of Salesforce that came in by acquisition and things like that. Right? We see exactly how challenging it is when you don’t get to do it in a monolith. And we’re making progress in all those ways as well. But it really does make us appreciate the elegance of the monolith that we do have. For all its warts, it is extremely powerful and extremely useful to our customers.

Josh Birk:

Right. Well, so let’s talk about that for a second, because as we’ve joked, this is a company that does a lot of acquisitions. It does a lot of acquisitions across industry and technologies. So how has that looked in the past sometimes, because I would imagine a product like Heroku is much different than say, Community Cloud or Commerce Cloud when it comes to trying to have that integrated back into the platform.

Ian Varley:

Yeah. I mean, it’s all over the map. I’m not an expert on acquisition, so I won’t speak to the strategy of it too much, but there are certain acquisitions that we do that just are tuck-ins right. That just become part of the product and get integrated very, very quickly. A recent example of that is actually Salesforce Maps. That was an acquisition called MapAnything. And I don’t think I’m giving anything away that’s not publicly known by saying that. And MapAnything was an incredible company that does really innovative stuff.

Ian Varley:

I mean, I think most people think like, oh, you’ve got Google Maps, and Google Maps is all you need to know, right? Everybody could just use Google Maps, or another similar service. But it turns out that in the enterprise world, when you’re trying to run a business, there’s all kinds of things that are not supported by sort of a consumer level map program like that.

Ian Varley:

And just an example of this is if you’re trying to ship things, you need to know clearance, right? How tall are your trucks, and how tall are those bridges that you’re trying to go under? And that’s not in Google Maps, right? You need something that’s a little bit more heavy duty to be able to give you routing and optimization with those kinds of constraints. That’s a bit of a tangent. MapAnything is a super cool company that now is completely integrated into Salesforce as Salesforce Maps. It’s not standalone in any way. It’s just its own as part of Salesforce now.

Josh Birk:

Yeah. And I have to just say, I’m actually quite glad for that evolution. Because back in the day, when everybody was doing demos around Google Maps, I always had to put in the asterisk, read their license. It’s not as open as you might think.

Ian Varley:

Yes, yes. Read the fine print for sure. And so that’s one example, but then there are absolutely like Heroku is a very, very different beast, right? Where Heroku is a completely independent sort of way of thinking about your development cycle. And there’s plenty of people who use Heroku who don’t use anything else from Salesforce. Now that said, Heroku is also a big part of the actual Salesforce strategy. And there’s stuff that we just announced at TrialheaDX about that integration that’s really interesting and cool. And so it is very integral. But for many years, it was completely standalone, and it was not integrated with the rest of Salesforce at all. So you have some that are like that.

Ian Varley:

And then there’s some that are kind of in between where it’s like, yeah, this is on the road to being fully integrated, but it’s not fully integrated yet. And I’m sure anyone who’s in the technology world can appreciate the complexity of taking two completely different architectures and merging them into one unified thing. It’s not child’s play.

Josh Birk:

You can’t just shove the two things together.

Ian Varley:

No, you get bad results if you do that. So there really is a pretty wide spectrum of how these things work. But the cool part about it is I guess that every acquisition is an infusion, not just of new, business capabilities and code and things like that, but also new patterns and new ways of thinking about things, right? When you’re in a company, and Salesforce is a fantastic company to work for, which means that lots of people stay at Salesforce for a very long time, which means they haven’t worked at other companies for a very long time. Right?

Ian Varley:

And so you tend very naturally towards being a little insular, and saying things like, “Well, that’s just how we’ve always done things.” And so it is really important to kind of get that infusion of new ideas, both from new hires, but also really collectively from joining with an entirely different software development culture or a different approach to things. And so that’s been hugely influential. And I mean, I think about it a little bit it’s sort of the melting pot idea of we get the best of what everybody is thinking across all these different acquisitions. And that’s important.

Josh Birk:

Nice. Okay. So let’s go back to core itself, what you and I think of as the application/platform. How would you actually define Salesforce core itself?

Ian Varley:

So fundamentally, Salesforce core is sort of a database with forms on top. I’m being a bit facetious, but not very right. When I first came to Salesforce, I saw that. And I was like, “Wait, isn’t this just database backed forms?” And they said, “Yeah, databased back forms, multitenant, as a service. That’s what people need.” Now what’s facetious about that is, of course, there’s actually tons and tons more sort of functionality value that’s layered on top of that. So I’m being facetious. There’s an immense amount of actual business tying product there.

Ian Varley:

But architecturally, when you really get deep down to the bottom of it, what is Salesforce? Well, the Salesforce platform is a way for people to define important business concepts and set that up to collect and view data about those important business concepts. And those concepts could be the ones that are built in as part of a particular product like accounts and contacts, are part of Sales Cloud, right? They could be, on the other end of the spectrum, things that are completely special to your business, right? If you sell golf clubs, then you could have a golf club object in Salesforce, right?

Ian Varley:

And what the Salesforce software does is it sets up all of the things that you need to semantically define and use this set of things via metadata. Now metadata, it’s a big word that has lots of different meanings for lots of different people. So I’ll tell you what I mean by metadata. For me, metadata is data that can alter the behavior of your software. Now that’s not everybody’s definition meditative, but that’s my definition of metadata. Right?

Ian Varley:

So for example, a configuration setting that tells you whether to display your charts in high contrast or not in high contrast for people with visual impairment, right? That’s a piece of metadata. That’s a configuration. Similarly, if you sell golf clubs, and the things that you want to store about a golf club are it’s make and model and color and size. And sorry, I don’t play golf. I don’t know what else you would define about a golf club, but that’s the schema of your golf club object. And that’s metadata in the system. Right?

Ian Varley:

But then beyond that, each of those pieces of information about that golf club, like the fact that the length is a number, right, that’s metadata as well. And the fact that a field in a object might be a URL, or it might be a date, right? Those contribute to the metadata that the system is doing. And then the software, what it does is it uses all of this metadata about your business, about your semantics to drive all kinds of fancy, cool stuff like generating reports in Tableau and allowing you to quickly mock up UI forms or automatically generate mobile views.

Ian Varley:

And so, because of the richness of the metadata, that’s what really gives the CRM core application its power to do all this amazing stuff. And then what you layer on top of that is then, yes, Salesforce engineers can define some of this metadata for the products that we ship like Sales Cloud, and service cloud. You, as an end customer, can define this for your own company, for your own org or tenant. Right? But then ISVs can also come in and say, :Hey, we’re going to offer a package that other customers could buy on top of Salesforce. And we’re going to define that metadata in a way that can be installed.”

Ian Varley:

So there is really this wide set of ways that the metadata can be used and can come about. But if I had to really describe what is this application, that’s what it is. It’s something that facilitates the creation and use of semantic and structural metadata to empower a customer to store data about the concepts that are important to them.

Josh Birk:

See, that’s interesting because when I’ve had people in the past ask me, defining standard objects and custom objects and metadata, I typically stick to a very structural point of view. In fact, I think the analogy I usually the use was like a blueprint. If the standard objects is your four room house, but you wanted to add a swimming pool. Well, the first thing we do is we actually extend the blueprint. We don’t just dig a hole in your backyard, and then be done with it. And the reason we have that blueprint is because if somebody else wanted that exact same swimming pool, we just moved the blueprint over to that house, and wash, rinse, repeat.

Josh Birk:

But you’re kind of saying that it’s a layer above that as well. So it’s not just the size and the shape of the pool, but also what kind of water they would use, what month that was going to be placed, and the architect that was going to be used. And the data around that data structure in order to kind of make the swimming pool really work.

Ian Varley:

And I’ll give you an example why that’s so important is if you think about Einstein and the AI use cases within Salesforce and the things we can do, a lot of what we can do there, we can do because we know more about this data. And let me qualify it in a second, but we know more about this data than just the kind of basic types you would see in a programming language like Java. We don’t just know that this is a number. We know that this is a phone number, right? And from an AI perspective, that tells us very different things about what we can do. Whether this is a useful thing to cluster data on. Which a phone number, it almost never is. Right? We have that kind of additional level of semantic understanding.

Ian Varley:

Now when I say we, just to be clear, what I don’t mean there is that Salesforce employees and engineers are going and looking at these things that customers do, and then creating AI on top of that. In certain situations, in a proof of concept where we have a special agreement with the customer, we’ll do that. But generally speaking, we don’t do any of that. This is all automated. We build the machines that can then introspect whatever the customer’s metadata is at runtime and do the appropriate thing precisely so that people from Salesforce don’t have to be sort of in a hands on way manipulating this to get good AI outcomes. All of that, that’s just one example of the kind of richness and value you get from having this deeper level of semantic integration around data.

Josh Birk:

Right. Well, let’s talk about how that really kind of works out physically. Because I remember back in my workshop days, people would ask me like, “Well, is all of this running on Oracle?” And my response was something like, “Well, kind of.” But not in the way you think. So how does this work from a physical database layer point of view in order to make what you just said actually happen?

Ian Varley:

Yeah. So one of the most interesting and maybe improbable things about the Salesforce software is how deep the metaness goes. I don’t know if that’s a word. Metaness.

Josh Birk:

I’ll allow it.

Ian Varley:

How deep the metaness goes. So when you create a custom object in Salesforce, under the covers, there is in fact a relational database down there that has rows for that. But what it doesn’t have is it does not have an actual physical table in the database for that object. Everything is done completely virtually. So under the covers, of course, there is a relational database there that’s storing all this stuff. But when you define a custom object, it’s not actually instantiating a table at the relational database level. It’s not doing a create table statement for real.

Ian Varley:

What it’s doing is just adjusting some metadata in a few tables. And then the entire engine of the application server is there to give the illusion that that’s a physical table. When in reality, custom object data is actually stored in one big table, one big relational database table called Custom Metadata. And it has, I think at last count, 500 fields, 500 varchar fields in it. And when I create an object called golf club, and I say that it has a length field, there’s a mapping in metadata that says, okay, that’s actually column three for this tenant. Column three, when you see one of these objects, column three is length.

Ian Varley:

And it does all of that translation in the layers of this application. So that you get back something that looks pretty much just like a results that might from doing a direct SQL query against the database. But it is indeed much more complicated than that. And the outcome of this, by the way, when I first started at Salesforce, and I saw that arrangement, I was like, no way. There’s no way that will ever work. It’s impossible. You can never run a big system like that.” And I was completely wrong. And in point of fact, there are a lot of things that we do to make that possible at the optimization level. We have a whole team dedicated full time to ensuring that the optimizer for our product is really working and working well.

Josh Birk:

Interesting.

Ian Varley:

I mean, if you’ve ever done anything with database internals, you know that generating the physical plan from a SQL statement is not trivial, especially when you have things like joins and all that stuff, right? You have to figure out what join strategies should I use. And if we were just to leave that up to the database itself to sort of figure that stuff out, nothing would ever function.

Ian Varley:

And so instead what we do is every single query that actually goes to the physical database is heavily, heavily hinted. We essentially exactly specify the query plan. We exactly specify, okay, if you’re doing a join, start with this, do a nested loop. Join to this other table, lean on this index. All of that stuff is exactly specified by the engine. Otherwise it wouldn’t ever work. So you really have this really interesting meta database. But then what that means is because the database is under our control, it’s got pluses and minuses.

Ian Varley:

Pluses are, we can extend it to do all kinds of really neat things that you couldn’t do directly in a relational database. The downside is that we pretty much are responsible for all of the things that a relational database would normally do out of the box for you. So for an example of this is indexes. We can’t just use indexes in the database. We have to maintain indexes and hints directly. Another one is foreign keys. We don’t have any foreign keys in the database. All of that has to be maintained at the software level because it’s so complex in this meta way. So it’s super interesting, super different from what I think people envision is happening, but it’s what has to happen because specifically it is a multitenant environment.

Josh Birk:

All right. Well, I want to talk about a couple ancillary topics. And one is one that I think you’ve admitted you could talk about all day long if you were allowed to. We don’t have that kind of time. So if you could give me the elevator pitch for just kind of simply why awareness of cognitive bias is important to your field of work.

Ian Varley:

Yeah. For those of you who are not super familiar, when I say cognitive bias, I don’t mean prejudice in favor of against a person or thing or group, as people often use the word bias. I just mean a systematic tendency to deviate from an expected value. That’s the kind of bias that, I mean. Like the engineering concept of bias like in an electrical system. A cognitive bias is a systematic pattern of deviation from rationality in your judgments. Basically it’s just bugs in our mental operating system, right?

Ian Varley:

A case where if you really sit down and think about something and do a logical analysis, you will come to one conclusion. But your brain, when push comes to shove, will not come to that conclusion, it will come to a different conclusion, right? And there’s a whole bunch of documented cognitive biases in the psychology literature. But when you think about these, so I did a presentation about this, about cognitive biases and software architecture, it was called Brains Considered Harmful. And it sort of just evaluated a bunch of different cases where our brains do the wrong thing.

Ian Varley:

So a simple example of this is the planning fallacy. The planning fallacy says that you basically always underestimate how long something’s going to take. It doesn’t matter where you start, it doesn’t matter what your prediction is, it doesn’t matter how you do it. You’re going to underestimate. And this is famous in software circles for the last 10% of the project takes 90% of the time.

Josh Birk:

Right. Exactly.

Ian Varley:

And so, we’ve learned as an industry to get past this using agile techniques and things like that. And there are other ways to do it. But my favorite example of this is actually called Hofstadter’s law, if you know, Douglas Hofstadter. He says, “Things always take longer than you expect them to, even when you take into in account Hofstadter’s Law.”

Josh Birk:

Nice.

Ian Varley:

So that’s just sort of a garden variety example of the kind of cognitive biases that can impact the creation of software. A more maybe nuanced one is one that’s near and dear to my heart called the sunk cost fallacy. And the sunk cost fallacy basically says we are sort of wired to throw good money after bad. If you’ve invested anything in some project or some idea, you will be unable to judge whether it’s the right thing to do because you’ve already invested in it. You will overvalue it. And this happens all the time to people who are doing software architecture. Because the sheer act of working on a technical system biases you in favor of that system, regardless of the facts of the matter.

Josh Birk:

Okay. So I wanted to wrap with something relevant to the time that we’re recording it in. Salesforce was working with various governments to enable them to fight COVID-19. It’s a huge push for the company right now. What’s some success we’ve had at that, and kind of specifically, how is this architecture that we’re talking about, how does that make it more successful?

Ian Varley:

Yeah, absolutely. So one of the key things that everyone should be doing, not that necessarily everyone is doing it, but that everyone should be doing during this period of our lives is something called contact tracing. Which is when someone gets sick with a virus, you find out as quickly as you can anyone that they’ve been in contact with, and you go and you contact those people and you tell them to quarantine. And if they haven’t been quarantining, you ask them who they’ve been in contact with, and you repeat.

Ian Varley:

And the whole idea here is to make people aware they’re infected soon enough that they can avoid infecting others, and thus, stop the spread of the pandemic, and reduce the R value and all that stuff. So this is this extremely basic, extremely critical thing that not everyone’s getting right right now. And one of the really cool things that Salesforce has done is worked with the state of Rhode Island to actually provide them the software, the software that helps them actually do the contact tracing, that backs this contact tracing information.

Ian Varley:

Because when you really think about it, this is not all that different from the kinds of information you would collect in a CRM system. This is facts about people, and where they were, and who they had relationships with. And it really sort of allows you to build up that picture. And Salesforce was able to do this precisely because we built a general purpose platform that allows you to collect any kind of data like this once you’ve defined it semantically.

Ian Varley:

So I think really in a very rapid way, Salesforce was able to produce contact tracing software, a subject that six months ago none of us had ever heard of, and to actually get it in the hands of, in this case, the state of Rhode Island who’s had immense success keeping COVID under wraps in in their state. So that’s just kind of a demonstration of, hey, the architecture of this product really lends itself to rapidly creating extremely valuable things that, if you had to start from scratch and build up all the infrastructure to do this, it would be very, very time consuming and costly.

Ian Varley:

And remember, this is on top of a platform where privacy and security are paramount concerns. This is enterprise software. This isn’t something that you just found on the internet and downloaded. This is very high grade, serious stuff that’s compliant and secure, and all this stuff. And so people are worried and should be rightly worried about the privacy of information that’s used in contact tracing systems. I don’t want that going into some just some shareware that somebody downloaded and installed on their laptop. I want that going into a system that that is verifiably and auditably secure And private.

Josh Birk:

And that’s our show. Now I do want to point out that in addition to the software engineering master’s degree and the philosophy degree you’d mentioned an earlier in this episode, Ian also has a music degree.

Ian Varley:

I actually play a lot of, these days, traditional Irish music, and I play accordion.

Josh Birk:

Oh, wow.

Ian Varley:

Yeah. That’s the right response to that. Oh wow.

Josh Birk:

I want to thank Ian for the great conversation. Of course, I want to thank you for listening. Now, we will include a link to Ian’s presentation on cognitive bias in the show notes. And speaking of those notes, if you want to learn more about this podcast, head on over to developer.salesforce.com/podcast, where you can hear old episodes, read the show notes, and also have links to your favorite podcast service. Thanks again, and I’ll talk to you next week.