If you joined us last week, you met Ian Varley. Ian is a Principal Architect here at Salesforce who has extensive experience with Salesforce CRM and multi-tenant architecture. He also comes from a musical background which has helped him develop his coding skills.

Last week, you heard part one of our conversation on Salesforce architecture. We split our discussion up so you could listen to it in full. Today in part two we talk all about metadata and why it is so important.  We also get into one of Ian’s favorite topics – cognitive bias in technology.

Tune in to finish the extended version of my discussion with Ian Varley.

Show Highlights:

  • The value of deeper semantic integrations of data.
  • How meta-databases physically work.
  • How unique IDs are generated.
  • What an instance is.
  • Why some IDs are 15 characters and some are 18.
  • Why awareness of cognitive bias is important.
  • The software Salesforce created to help stop the spread of COVID-19.

Links:

Episode Transcript

Ian Varley: And I’ll give you an example why that’s so important is if you think about Einstein and the AI use cases within Salesforce and the things we can do, a lot of what we can do there, we can do because we know more about this data.

Josh Birk: That is Ian Varley, principle architect here at Salesforce. I’m Josh Birk, your host with the Salesforce Developer Podcast. And here on the podcast, you’ll hear stories and insights from developers for developers. Today, we sit down and talk with Ian in our second part of our extended interview. This is effectively the original interview from the one that we released last year. Here, we are talking about metadata, and we’re just going to get back to Ian and talking about the importance of it.

Ian Varley: Let me qualify it in a second, but we know more about this data than just the basic types you would see in a programming language like Java. We don’t just know that this is a number, we know that this is a phone number. And from an AI perspective that tells us very different things about what we can do, whether this is a useful thing to cluster data on, which in phone number it almost never is. We have that kind of additional level of semantic understanding. When I say we, just to be clear, what I don’t mean there is that Salesforce employees and engineers are going and looking at these things that customers do and then creating AI on top of that, in certain situations, in like a proof of concept where we have a special agreement with the customer we’ll do that, but generally speaking, we don’t do any of that, this is all automated.
We build the machines that can then introspect whatever the customer’s metadata is at runtime and do the appropriate thing precisely so that people from Salesforce don’t have to be in a hands on way manipulating this to get good AI outcomes. That’s just one example of the kind of richness and value you get from having this deeper level of semantic integration around data.

Josh Birk: Well, let’s talk about how that really works out physically, because I remember back at my workshop days, people had asked, “Well, is all of this running on Oracle?” And my response was something like, “Well, kind of, but not in the way you think.” So how does this work from a physical database layer point of view in order to make what you just said actually happen?

Ian Varley: So one of the most interesting and maybe improbable things about the Salesforce software is that how deep the metaness goes, I don’t know if that’s a word, metaness, but…

Josh Birk: I’ll allow it.

Ian Varley: How deep the metaness goes. So when you create a custom object in Salesforce, under the covers there is in fact a relational database down there that has rows for that, but what it doesn’t have is it does not have an actual physical table in the database for that object. Everything is done completely virtually. So under the covers, of course, there is a relational database there that’s storing all this stuff, but when you define a custom object, it’s not actually instantiating a table at the relational database level, it’s not doing a create table statement for real, what it’s doing is just adjusting some metadata in a few tables and then the entire engine of the application server is there to give the illusion that’s a physical table. When in reality, custom object data is actually stored in one big table, one big relational database table called custom entity data.
And it has, I think at last count, 500 fields, 500 VaR chart fields in it. And when I create an object called golf club, and I say that it has a length field, there’s a mapping in metadata that says, “Okay, that’s actually column three for this tenant.” Column three in the custom entity, when you see one of these objects, column three is length. And it does all of that translation in the layers of this application so that you get back something that looks pretty much just like a result set might from doing a direct sequel query against a database, but it is indeed much more complicated than that. By the way, when I first started at Salesforce and I saw that arrangement I was like, “No way, there’s no way that could ever work. It’s impossible. You could never run a big system like that.” And I was completely wrong.
And in point of fact, there are a lot of things that we do to make that possible at the optimization level. We have a whole team dedicated, full time to ensuring that the optimizer for our product is really working, working well, and if you’ve ever done anything with database internals you know that generating the physical plan from a sequel statement is not trivial, especially when you have things like joins and all that stuff, you have to figure out which joins strategy should I use. And if we were just to leave that up to the database itself to figure that stuff out, nothing would ever function. And so instead what we do is every single query that actually goes to the physical database is heavily, heavily hinted. We essentially exactly specify the query plan. We exactly specify, okay, if you’re doing a join, start with this, do a nested loop, join to this other table, lead on this index, all of that stuff is exactly specified by the engine otherwise it would never work.
So you really have this really interesting meta database, but then what that means is because the database is under our control, it’s got pluses and minuses. The pluses are we can extend it to do all kinds of really neat things that you couldn’t do directly in a relational database. The downside is that we pretty much are responsible for all of the things that a relational database would normally do out of the box for you. So for example of this is indexes, we can’t just use indexes in the database, we have to maintain indexes enhanced directly. Another one is foreign keys. We don’t have any foreign keys in the database. All of that has to be maintained at the software level because it’s so complex in this meta way. So it’s super interesting, super different from what I think people envision is happening, but it’s what has to happen because specifically it is a multi-tenant environment.

Josh Birk: Well, and also it’s a multi-tenant environment with a lot of abstractions to it. Like the swimming pool, you need able to move that person’s definition of their version of our application potentially around 3, 4, 500 orgs. So let’s talk a little bit about the magic of that idea itself. So we’re creating our own primary key, foreign key system, and I think a lot of people look at that Salesforce ID that many of us, especially developers, have stared at over and over again and we probably don’t think about it much other than the fact that, well, Salesforce is generating a unique ID for me. What else is happening under the hood to make that ID work?

Ian Varley: As you probably have sussed out, if you’ve looked at IDs, there is structure inside the ID. It’s not just a unique value. So the first three characters of that ID are what’s called the key prefix. And the key prefix tells us what kind of thing is this. So a user record, if you go and look on a user record, is going to have I think that’s 005, and a chatter post I think is 0D5. It’s sad that I have [inaudible 00:07:03] and maybe I’m getting them wrong, in which case I’ll look silly, but each of these has maps to some semantic concept in the system and of course then when you create custom entity’s they get prefixes as well. So then within that there’s some amount of uniqueness of that, but then the next part of the key, right after the key prefix, is what we call the instant ID.
That’s the character that follows that immediately. And actually I should back up a second. So the space of possible values for each digit in an ID is it’s going to be a set of characters that includes lowercase letters, uppercase letters, and the digits zero through nine. So if you do math, that is 62 possible character values. Now, why they didn’t just add two more random characters and make it base 64? I don’t know. That seems maybe no one thought about it too much, but would’ve been some nice, more efficient representations we could have used if they had done that, but it’s base 62. And so you can think of the idea as really a 15 digit base 62 number, which means that it is a vast, vast number, the set of all possible spaces there.
So, as I said, the first of the three key prefix, the two characters after that are what’s called the instance ID. I guess I should back up a little bit and talk about what an instance is just for a second. So I mentioned that there’s the application servers in the database, but there’s not just one of these. There are actually hundreds of these pairs of a set of application servers in a database. And in the beginning there was just one, but then as we grew we realized we would not be able to support the entire world using Salesforce on just a single database even if we got a really big one, which we did, and so we split it from one to two and then we split it from two to three.
And when I say we split it, we essentially chard the set of tenants or orgs on that instance across multiple of these instances, multiple of these replicas of the hardware stack, so any customer lives on one or the other, but not both. And so we’ve scaled out that way and now we have hundreds of these instances. And so the reason that there is a unique couple of characters there in the idea itself that tells you the instance you’re on is to avoid key collisions. Now, a 15 character basics two number is a very, very large number, but it is not infinite and it’s entirely possible, and especially if you’re doing sequential ID generation, that you would end up with conflicts where you have the same idea on multiple instances.
So this is essentially just a way to ensure that never happens because the oracle of creating these IDs lives on each individual instance and so it doesn’t have to ask all of the other instances, hey, has this ID been used before? It can just keep track of it itself and then it just disambiguates it with that instance. Now, I should caveat that by saying that organizations or tenants do move around between instances for various reasons, it’s called a migration, and so those two characters don’t actually tell you the instance you’re currently on, they just tell you the instance where the record was created and that’s an important distinction.

Josh Birk: Got it.

Ian Varley: And so from there, there’s a couple of reserved characters, which I can tell you more about, but then the rest of it is a big open, unique number for this particular record.

Josh Birk: Got it. And one of my favorite trivia questions of all time, why are some IDs 15 characters and why are some 18?

Ian Varley: Oh, isn’t that a fun one? So remember what I said about the space of these 62 different characters, it includes uppercase and lowercase letters. And so a record with an uppercase A is a different record from a record with a lowercase A in the same position, even if everything else is the same, which is all well and good for databases, which are pretty good at keeping case sensitivity. And we know in our application that we built, we said, “Yeah, we can make it all case sensitive, and that’s great.” It turns out however, and I don’t actually know if this is the case anymore, but I believe there were certain… I can’t remember if it was browsers or API clients, that did not do a very good job of ensuring case sensitivity. And so you’d be going to request this and somewhere in the pipeline as part of your request it would say, “Oh, it didn’t matter whether that was a lowercase or an uppercase A, just send it as an uppercase A.
Which I’m sure there was a very good reason for some programmer somewhere to have done that, but it caused us a lot of grief. So the nice solution to that is the idea of a check sum, and what a check sum is it just says, “Okay, I’m going to look at the all the digits in this ID and I’m going to produce a little unique value that just tells you what is the case of each of the values along the way here, and we can compactly represent that in just three characters and tag it onto the end of the ID.”
So basically what that means is even if it came through with the case all mangled, you could use those three characters on the end to tell you what the actual case of the other letters in the ID was, and so this little checksum basically means we now no longer care whether that happens or not, and so that’s the difference. If it has the checksum, it’s 18 characters. If it doesn’t have the checksum, it’s 15 characters. It’s the same idea in either case, just one has the check sum and one doesn’t.

Josh Birk: Just one has a check. Got it. Okay. All right. Well, I want to talk about a couple ancillary topics. And one is one that I think you’ve admitted you could talk about all day long if you were allowed to, we don’t have that kind of time, so if you could give me the elevator pitch for just simply why awareness of cognitive bias is important to your field of work.

Ian Varley: Yeah. So for those of you who are not super familiar, when I say cognitive bias, I don’t mean prejudice in favor of or against a person or a thing or group as people often use the word bias, I just mean a systematic tendency to deviate from an expected value. That’s the kind of bias I mean, the engineering concept of bias like in an electrical system. A cognitive bias is a systematic pattern of deviation from rationality in your judgments, basically it’s just bugs in our mental operating system. A case where if you really sit down and think about something and do a logical analysis, you will come to one conclusion, but your brain, when push comes to shove, will not come to that conclusion, it will come to a different conclusion.
And there’s a whole bunch of documented cognitive biases in the psychology literature, so I did a presentation about this, about cognitive biases and software architecture, it was called brains considered harmful and it just evaluated a bunch of different cases where our brains do the wrong thing. So a simple example of this is the planning fallacy. The planning fallacy says that you basically always underestimate how long something’s going to take. Doesn’t matter where you start, doesn’t matter what your prediction is, doesn’t matter how you do it, you’re going to underestimate. And this is famous and in software circles for the last 10% of the project, it takes 90% of the time.

Josh Birk: Exactly.

Ian Varley: And so we’ve learned as an industry to get past this using agile techniques and things like that, and there are other ways to do it. My favorite example of this is actually called Hofstadter’s law, if you know Douglas Hofstadter, he says, “Things always take longer than you expect them to, even when you take into account Hoffsteader’s law.” It’s just [inaudible 00:14:22]. So that’s just a garden variety example of the kind of cognitive biases that can impact the creation of software. A more maybe nuanced is one near and dear to my heart called the sunk cost fallacy, and the sunk cost fallacy basically says, “We are wired to throw good money after bad.” If you’ve invested anything in some project or some idea, you will be unable to judge whether it’s the right thing to do because you’ve already invested in it, you will overvalue it.
And this happens all the time to people who are doing software architecture, because the sheer act of working on a technical system biases you in favor of that system, regardless of the facts of the matter, and we just don’t throw things away as much as we probably should.

Josh Birk: Back in my consulting days, I would refer to that as asbestos where it’s like that’s the stuff that’s in your walls and you think it’s keeping you safe and warm, and it seems like it’s going to be so much work to get it out of that wall and replace it with something better, but it’s also killing you.

Ian Varley: Exactly. An awareness of cognitive biases as a general concept I think is really useful for software architects, I think just knowing really is, in many cases, half the battle and so just adjusting for it is a good place to start. And I think I can give you a link to that actual presentation. Let me see if this is the right one.

Josh Birk: You presented that at the O’Reilly conference, right?

Ian Varley: I did. Yes. At the O’Reilly Software Architecture Conference. And if you just go to bitly.com/brains-considered-harmful, you can find it.

Josh Birk: Nice.

Ian Varley: Sorry. Not, billy.com. Bit, L-Y.

Josh Birk: Got it. And we will definitely also include that in the show notes for anybody who’s listening. Okay. So I wanted to wrap with something relevant to the time that we’re recording, and Salesforce was working with various governments to enable them to fight COVID-19. It’s a huge push for the company right now. What’s some success we’ve had at that, and specifically, this architecture that we’re talking about, how does that make it more successful?

Ian Varley: Absolutely. One of the key things that everyone should be doing, not that necessarily everyone is doing it, but that everyone should be doing during this period of our lives is something called contact tracing, which is when someone gets sick with a virus, you find out, as quickly as you can, anyone that they’ve been in contact with and you go and you contact those people and you tell them to quarantine. And if they haven’t been quarantining, you ask them who they’ve been in contact with and you repeat. And the whole idea here is to make people aware they’re infected soon enough that they can avoid infecting others and thus stop the spread of the pandemic and reduce the R value and all that stuff. So this is this extremely basic, extremely critical thing that not everyone’s getting right right now. And one of the really cool things that Salesforce has done is worked with the state of Rhode Island to actually provide them the software, because the software that helps them actually do the contact tracing, that backs this contact tracing information.
Because when you really think about it, this is not all that different from the kinds of information you would collect in a CRM system. This is facts about people and where they were and who they had relationships with, and it really allows you to build up that picture, and Salesforce was able to do this precisely because we built a general purpose platform that allows you to collect any kind of data like this once you’ve defined it semantically. So I think really in a very rapid way, Salesforce was able to produce contact tracing software, a subject that six months ago none of us had ever heard of, and to actually get it in the hands of, in this case, the state of Rhode Island who’s had immense success keeping COVID under wraps in their state.
So that’s just a demonstration of, hey, the architecture of this product really lends itself to rapidly creating extremely valuable things that if you had to start from scratch and build up all the infrastructure to do this, it would be very, very time consuming and costly. And remember, this is on top of a platform where privacy and security are paramount concerns. This is enterprise software. This isn’t something that you just found on the internet and downloaded. This is very high grade, serious stuff that’s compliant and secure and all this stuff. And so people are worried and should be rightly worried about the privacy of information that’s used in contact tracing systems. I don’t want that going into just some shareware that somebody downloaded and installed on their laptop, I want that going into a system that is verifiably and auditably secure and private.

Josh Birk: Because going back to trust is our number one value and also there’s a whole layer of the Salesforce kernel we didn’t even get a chance to talk about in the terms of we’re talking about forms and databases, but there’s also access and security.

Ian Varley: Exactly. And the ability to rigorously define and enforce that is why people turn to Salesforce for this, as opposed to just throwing it in a spreadsheet or something. I’m very proud to say that I think Salesforce has done an amazing job in providing software to do this. Now, what I wish is that more places in the world, in particular places like the place that I live, like Texas saw the value in doing this kind of work and could use this software.

Josh Birk: And that’s our show. And before we go, I did ask after, Ian’s favorite non-technical hobby and well, it’s one that’s come up on the pod a lot because apparently programmers and musicians share a similar kind of brain.

Ian Varley: I am a musician. We were talking earlier about my music degree, I do still play music. I actually play a lot of, these days, traditional Irish music and I play accordion really.

Josh Birk: Oh, wow.

Ian Varley: That’s the right response to that, oh, wow. But no, I love it. It’s a fascinating hobby, and actually it’s totally something you can geek out about in that the way that traditional Irish music works is people get together in a pub, or these days on Zoom, and sit around and trade tunes, but there’s this immense corpus of tens of thousands of these very specific melodies, these tunes that you have to learn, and if you don’t know the tune, you can’t just jam along to it, you either know it or you don’t. So it’s this immense memorization exercise effectively that you have to do to just be able to play at a basic level with people, so it’s very nerdy from that perspective so that’s part of why I like it.

Josh Birk: I want to thank you, Ian, for the great conversation and information. And as always, I want to thank you for listening. Now, if you want to learn more about this show, head on over to developer.salesforce.com/podcast where you can hear old episodes. See the show notes that links to your favorite podcast service. Thanks again everybody, and I’ll talk to you next week.

 Get notified of new episodes with the new Salesforce Developers Slack app.