Nikita Prokopev In this episode, we explore the technological journey of Nikita Prokopev, from his early fascination with DOS and PC games to his growth into a seasoned programmer. Listen in as Nikita reveals his transition into the Salesforce ecosystem, the evolution of the Salesforce platform, and how it shaped his professional trajectory.

Discover his motivations behind taking the entrepreneurial plunge and how he juggled multiple roles to make his venture a success. He underscores the importance of observability in the Salesforce platform and shares insights on how it can be utilized to improve performance and efficiency.

Nikita breaks down observability into various components and explains how it helps monitor the health of a system and prevent errors. Also, get valuable tips on how to maximize logging efficiency and performance to control data volume and avoid hitting platform limits. Tune in for a fascinating exploration of Nikita’s journey in technology and valuable insights on Salesforce.

Show Highlights:

  • The challenges he faced in the early days of Salesforce.
  • The importance of observability in Salesforce.
  • How observability can help monitor system health and prevent errors.
  • How to maximize logging efficiency and performance to control data volume and avoid hitting platform limits are shared.
  • The transition from a passion project to a professional career in technology.

Links:

Episode Transcripts

Nikita Prokopev:
And it was a blast. We had the PC speaker for sound that made those annoying noises. To me, that was just symphony. It was better than Beethoven, than anything. It was amazing, this machine.

Julián Duque:
That’s Nikita Prokopev, Founder, CEO, and Head of Product at Pharos AI. I’m Julián Duque, your host for the Salesforce Developer Podcast and here in the podcast, we share stories and insights from developers, for developers.
Today, we are going to talk with Nikita about observability, but before, we will start, just as we left off and we often do, with his early years.

Nikita Prokopev:
It’s an interesting question. I think I was about six years old and we got our first PC. My dad brought it, I don’t know where he got it, but he brought it for some word processing. And if my memory serves me right, this was at 286.

Julián Duque:
286?

Nikita Prokopev:
Yeah, 286. Some years ago, you had nothing but the MS-DOS. I don’t think we even had a Norton commander on there. It was just all straight-up command line. And of course, having not much entertainment back in the day where I grew up, this was it. I mean, this was the holy grail for me. I fell in love with the thing the moment I set my fingers on the keyboard and I did everything. I learned to type. We had some games on there, those old DOS games with 16 colors and whatnot. It was a blast. We had the PC speaker for sound that made those annoying noises. To me, that was just symphony. It was better than Beethoven, than anything. It was amazing, this machine.
It went from there, I guess. Then we upgraded to 386, and we eventually got a Pentium, and on and on and on. And so the computer has been with me since the tender age of six.

Julián Duque:
Wow. It sounds very similar to my story. I remember having one of those CRT monitors with a bottom that you change from green to orange to white. Those were like the three different colors this monitor managed.

Nikita Prokopev:
Yeah, it was glorious. Some of my friends had the ZX Spectrum, the ones that you put the tapes into, and of course, you were able to rewind or fast-forward to find a game, and then you press play and it made those weird noises. And then when the tapes would get out of shape for whatever reason because they’re magnetic tapes, so they’re not eternal, there was so much frustration there. Just thinking back to it now, those were the good old days, and it just makes me feel old too, I guess.

Julián Duque:
Definitely. Of course. Not old, I mean. We had the privilege to be able to experience computers at a young age.

Nikita Prokopev:
Yeah, for sure.

Julián Duque:
What happened after that? What brought you into technology or what brought you into programming?

Nikita Prokopev:
Well, I think the main driver for me to learn how to code was the fact that you only could do so much with those old computers and the games that you had were far and few in between. So you exhausted all of them, and then at some point, you had to make your own entertainment. And that’s where it really started for me. That’s where I learned about QuickBASIC and eventually Turbo Pascal. And then from there, as Windows became a thing, it was Visual Basic and Visual C++, and all those things.
Once I went down that path, that was it. That’s where it started for me. All of my friends were into it. My dad’s a scientist, so he was naturally keen on fostering this hobby. And so we bought all kinds of books, how to code, this and that. And some of them were a little too far out there for me, but I learned a lot from my friends, especially older friends. They were taking some computer classes at school. We had a pretty advanced school for the place and the age that I grew up in. And so they taught me a few things.
And then yeah, my aspiration was always to make games, as weird as it sounds, and I’ve tried. I’ve made card games, really primitive ones like blackjack and poker and all those things. I always wanted to make a 3D engine Doom-like games. When I saw Doom, that was just the thing for me, and I thought, “There’s nothing better than this. This is it. This is where progress of humanity stops. And at that point, it doesn’t get any better.”

Julián Duque:
Well, you see today, it keeps getting better, which is very impressive.

Nikita Prokopev:
Oh, absolutely, yeah. But it is funny. As a kid, you don’t think that way. You go onto the next best thing and the next best thing and everything is the best thing. And of course, after Doom, there was all the fancy 3D graphics that started coming out, Quake and that line of games, and software and Half-Life. And I was always a big gamer back in the day, so this was very much hand in hand with programming for me. It was why I learned programming.
And then later on when I got to college, that’s where things took off in a bit of a different direction. I learned programming and the concepts of computer science, and it really sparked my interest in a different way. I’m like, “Wow, this is interesting.” I always liked math, and so this was a math on another level. This is like, “Wow, I learn about complexity and algorithms and running time and all these different things,” and it just went down that black hole.

Julián Duque:
Wow. For me, it’s the ability of creating something.

Nikita Prokopev:
I agree.

Julián Duque:
You are using certain tools and you are creating something and seeing that manifestation is… For me, that’s powerful. It’s what made me get into this.

Nikita Prokopev:
Yeah, I agree 100%. And the fact that you don’t even need anything physical for it was the other thing. You could just sit there and stare at this monitor and lo and behold, something beautiful comes out and it just captivates you.

Julián Duque:
What are you most proud about from those times, something that you built or something that you learned? What is your most fondest memory?

Nikita Prokopev:
Yeah, so funny enough, so when I was growing up, computers were expensive and I never really had a good one. And when I immigrated to the US at the tender age of 17, I got into a community college, and of course, I enrolled in the computer science program right away. And the nature of education that I got back in high school was such that it was a bit too advanced and not in a good way. They really pushed you. But I ended up with all this knowledge about math and science and whatnot. And in college, basically when I started off, I was able to proficiency out of a few math classes, calculus, and some physics classes that I had to have for my degree, but I didn’t have to take them. I just took the tests and then I was free to do whatever I wanted for the semester.
And on top of that, I got a tutoring job, tutoring math at the community college. And they had a lab. It was a big computer lab, maybe 20 machines there, and they weren’t great machines, but it’s better than anything I’ve ever had. And so I would just hop from one computer to the other, find the best one that I like, and I’d install, I think it was Turbo C++. And I had a book with me. It was the Bjarne Stroustrup book, the classic.

Julián Duque:
The bible.

Nikita Prokopev:
Learned C++. Yeah, the bible. And I started reading it. I’m like, “Okay, well I know Pascal, that’s no fun. I hear cool kids aren’t coding in that anymore. It’s all about C++, so let me give that a shot.” So I sat down and opened the book and started going chapter by chapter, “Oh, here’s structures, here’s classes, here’s object-oriented programming. Why don’t I try something?”
And so I built a Tetris game just playing around, and it was a pretty sweet Tetris game. I put some graphics into it. It really was a passion project, and I learned C++ along the way. So probably that moment was where I was like, “Okay, I have confidence that I can learn things.” And once I knew that fact, then from there, it was just a whole other level. I can learn anything, what’s next? Yeah, so that’s probably it.

Julián Duque:
That’s beautiful. And how was that transition from something like a passion project, a hobby, to working on this, something professional?

Nikita Prokopev:
A real job?

Julián Duque:
A real job. So your first job after tutoring was working as a software developer or how was that transition?

Nikita Prokopev:
Yeah, so my first job, I actually got a tech support gig at the same community college, but this wasn’t really programming. This was more like fixing computers, building computers. I’ve done that. That was pretty easy. But my first programming job was when I got to my junior year in college and I went to work for NCSA at the University of Illinois, which is the National Center for Supercomputing Applications. It was like a dream come true. The building was beautiful. They had a giant supercomputer in the basement, and I got hired there. My buddy was working there at the time, and so he put a word in for me.
And so I went up there and sat through an interview and I didn’t know what to do. I’ve never been in a real job interview. And so I just printed out a bunch of my code and I brought it with me. And so I handed off a stack of papers to the guy and he looks at me and he starts going through sheet by sheet and like, “Okay, well, let me ask you some questions.” And I was super nervous. I don’t know how I got that job, but I did.
And the job was in Java, which I had no idea how to write in Java because being used to C++, and you might think it’s similar, which it is, but there are very big differences, even back in the day when Java was really young. I still had to learn a lot of concepts and you had to learn all the IDs because I was used to your standard either Linux environment or a UNIX environment with Emacs or something like a Visual Studio ID. And in Java, you had totally different technologies, totally different tools, and it was rough. I was so stressed out. I wanted to do good, and I had a real project at hand with the requirements. I had to build, what was it they had me build? It was like a SourceForge. I don’t know if you remember what SourceForge is?

Julián Duque:
Yes.

Nikita Prokopev:
Yeah. It’s like a repository of code. Back in the old days before GitHub, you had that kind of a public-facing knowledge repository and it’s all open source. You can go download code and play around with it and whatnot. So they had me build a SourceForge for astronomy scripts and various bits of logic, and they were… Yeah, it was a web application using a framework. It was a homegrown framework. I had no idea what a framework was. It was like, “Oh my God, this is so complicated. I don’t understand any of this. There’s XML files. I got to start”-

Julián Duque:
I imagine using Servlets, JSP.

Nikita Prokopev:
Oh, yeah. Yes, exactly. It was JSP-based. It was running on, oh gosh, what was it? Tomcat, Apache Tomcat.

Julián Duque:
Tomcat, yeah.

Nikita Prokopev:
Old version of Tomcat. And to me, this was just a whole… I didn’t sleep for literally a week because I realized I don’t know anything about this stuff and it scared the heck out of me. So I tried to learn as much as I could. Eventually I felt comfortable, but it took me a while to get there. It was a very stressful time, but I’m glad I did. In retrospect, I really enjoyed that time.

Julián Duque:
Coming from the confidence of building a Tetris in C++ to intimidating Java.

Nikita Prokopev:
Oh, yeah. It was a whole other level. I’m like, “Yeah, I can do the graphics anytime,” and the game makes sense to me and I could build it however I want. But here, you had the framework, you had all these new rules, and there was a database. I think we were using a SQL server database. I have to learn SQL. I had to learn about stored procedures and joins and all these things that just I never realized existed out there. So it was a fun time.

Julián Duque:
Definitely. I really love these stories because everybody has a different path, but at the end, you get to the same point. And I want to ask you, how was that now transition into the Salesforce ecosystem? How you got into Salesforce.

Nikita Prokopev:
It’s a great question. So the university, having finished that junior year, I got an internship thanks to that NCSA job. I went and did that internship. It was like a three-months internship over on the East Coast. I came back and I started interviewing for getting ready to get out of college and go on to the next level. And my difficulty was that I was on the student visa and I had to find a company that would sponsor me. It wasn’t a matter of just pick any job and you’re good to go. I wasn’t authorized to work in the country unless a company would sponsor me. So my pool of potential jobs was limited, and so I had to go everywhere and ask everybody.
Interestingly enough, Salesforce was there at the career fair. It was their first-ever college recruiting program. Now, this was back in, I want to say ’05, somewhere in there. So I got hired through that program into Salesforce, and having done Java, I had that experience. I did pretty good on the interview. They flew me out and I did the in-person interviews, and that was it for me. From that point on, I joined Salesforce as an engineer full-time where I’ve been for about eight and a half years from that date. And I’ve learned about the ecosystem after that, which that’s the next phase of the journey.
But back then, Salesforce wasn’t on the map. It was certainly on the map in Silicon Valley here and there, but it wasn’t a global company that it’s today. And so yeah, it was interesting. I thought, “Oh man, nobody really knows the Salesforce thing, but then you’ve got Microsoft and you got Google and you got Amazon. Everybody knows them. What am I doing?” I just really liked San Francisco, so I wanted to move there, and that’s what I did.

Julián Duque:
Yeah, eight and a half years is quite the time. And I bet you learned a lot from the internals of Salesforce.

Nikita Prokopev:
Oh, it was very much the same as when I got that first programming job. I had to learn everything. I thought I knew something, but I didn’t know anything. And going through that journey once again, and of course later on, I realized that’s how it’s going to be for the whole career. You never stop learning. Yeah, I learned a ton. It was an incredible journey. I got a chance to be a part of all these different teams, work on all the different aspects of the app from the front end to the database layer to some of the security aspects.
One of my first projects, interestingly enough, was building out an internal CPQ app for Salesforce. That’s where when Apex and Visual Force had just come out, they weren’t even public yet. I think they were going to announce them at the upcoming Dreamforce, but they weren’t public yet. And so we were building on that stuff. There was no metadata API and deployment tool was just in the very, very early stages. That was a crazy world right there.

Julián Duque:
Well, gladly things have evolved and we have way better tooling now for that.

Nikita Prokopev:
Oh, yeah.

Julián Duque:
So now going a little bit further on your path, working at Salesforce or building Salesforce, and now you are building for Salesforce with your own company. How was that jump? Why you decided to go and start your own company?

Nikita Prokopev:
Yeah, that’s a great question. I suppose somewhere along the road of me being an engineer, I very much enjoyed the technical aspects of it and being able to work with amazing people and learning along the way. But somewhere along that road, I realized, hey, it’s not just code that I’m writing for the sake of code. This code is doing things. And having worked with product owners and product managers and some of the business folks, that side of it started to appeal to me more and more. I’m like, “Wow, that’s cool. This is what you need to accomplish, and this is how you do it, and this is what you need to get things done.”
So picking up some of the project management aspects, and it really appealed to me, the fact that I knew exactly what I was capable of doing by myself. I could sit there at the computer and write code all day, but then what happens when you have a team of people? And I knew firsthand what that was like. I was being part of that team. And then the next step from there was, hey, what if I had my own team? What could I do then?
And that’s where I started thinking about, “Hey, this platform is getting really mature. I know it really well.” Back in the day when I started doing consulting, there wasn’t a Trailblazer. There wasn’t really, I mean, I think certifications were a thing back then already, but there wasn’t a whole lot of knowledge about how to build on the platform. And so I figured, hey, I might as well go out and try and experience this and see what I can do.
And so that’s where I started my first services company with a friend of mine. He was the business guy, I was the tech guy as it goes frequently in these types of scenarios. And we went out and he found some customers for us. I did the coding, I was the project manager, the senior developer, the architect, the support engineer. It was cool being my own boss to some extent. Of course, I didn’t realize it back then that now you have customers and they are your boss and you have many of them instead of just one. But that’s where I think it happened for me.
And from there, I wouldn’t stop. It was a whole new level. Once again, when you come to a point where in your journey you reach that level, you have to keep exploring it and keep going until you’re either satisfied or there’s another level that you see, and so that’s how it went.

Julián Duque:
Yeah. Now bringing that into the present because this whole trajectory has been fantastic, I love the success story of folks that find a purpose and a business within the whole Salesforce ecosystem. And today, you are building or you already have a great product that solves one very important and a specific issue, which is observability.

Nikita Prokopev:
That’s right.

Julián Duque:
So tell me a little bit more about why observability is important or what is observability for the folks out there that don’t know about the term?

Nikita Prokopev:
Sure. Yeah, absolutely. I think in order to answer that question fully, I’ll take a step back to the Salesforce days for just a little bit, so bear with me.
When I first joined, mind you, the no software thing was not a thing at all. Everybody was so suspicious. “What do you mean you have my data? What do you mean I don’t own the data center? What are you going to do with my data?” It was a long road to prove to the world that this is the new way to do software. You didn’t have to do physical upgrades. It was a single delivery model. You push out a release, everyone gets the functionality all at once. And now, we don’t think much about this because there are so many tools out there that are like this. Everything’s in the cloud these days. So you go to your Gmail, you open it up, you don’t even know if they’ve made changes. Sometimes they tell you, sometimes they don’t, but they have.
And figuring out how to do that from scratch from an engineering standpoint was no small feat. You had to have a solid DevOps and a quality pipeline. And part of that pipeline, of course, you got your dev process, you got your unit testing, functional testing, you have your manual testing. There’s all sorts of testing that goes on, and it repeats on and on with each stage in the pipeline. And an important part of that, which comes at the very end, or sometimes it doesn’t have to be at the end, but it can also be incorporated in each of the pipeline, is that observability piece.
So what is the observability? To answer that original question, it’s simply understanding what’s under the hood. How are things functioning. When you fire up your car, you know how fast you’re going, you know if your check engine light’s on, you got your indicators telling you the status of the engine that’s under the hood. You’re not taking the engine apart and looking at it physically. You’ve got things that tell you if it’s working fine, if it’s doing what it’s supposed to, and then alerting you if something goes wrong. That is observability in the nutshell.
And if we want to go academic, it’s actually a concept from control theory, but it’s your logs, it’s your monitoring, visualizations, metrics, alerting, tracing, all of the stuff that you need to know about the runtime. And the runtime is what matters at the end of the day. If your app doesn’t run the way you intended it to, then what good is it? It’s fundamentally important to your business. There’s no other way around it.
That’s the problem that we’re trying to solve and I’ve seen that firsthand at Salesforce, how it was done. Every release we did GACK watching, making sure that there are no errors in the instance once we released a new code. That was extremely important because next Monday or the Monday after that weekend, customers would hit the instance. And if they find the problems before the engineers do, well, at that point, it’s all sorts of trouble. You got your reputational problems, you as a company, especially a company that’s paving that new software ideology, that’s the end of that. No one’s going to trust you and no one’s going to go for this whole idea. That, to me, was how software was supposed to be built.
Now, in the ecosystem, things were different back then. There was simply no process around this. There was no idea of DevOps, there was certainly nothing about observability. Yeah, you look out for problems when users complain, you go and fix stuff, but that’s all after the fact. Very reactive. That’s what we wanted to change with the product that we’ve built. It’s getting people more into the mindset of being proactive, monitoring things, being conscious of what their environment is experiencing, what their users are experiencing, and so on and so forth.

Julián Duque:
Yeah. I had the opportunity to attend one of your sessions at Forcelandia about observability, and you said something that caught my attention, that pretty much the main goal is to prevent rather than fix. That’s the proactivity.

Nikita Prokopev:
That’s correct. That’s exactly right.

Julián Duque:
Yeah. And you also gave an analogy, I love that analogy, if you can repeat it here for our audience, the fire danger analogy.

Nikita Prokopev:
Oh, I’ve done so many of these. I’m just like, “Okay, which analogy? Was it the fire one?”
Yeah, the fire is a good one. We all know this. Those folks that are out there in California or in the Pacific West, we deal with fires, wildfires on a seasonal basis, fire season’s pretty much all year-round these days. And it’s not just the West, it’s all over the country now. There were a bunch of fires in Canada not too long ago. I actually don’t know if they’re still going.
But yeah, once you have a fire going, it’s really difficult to stop it. It consumes everything in its path, and you can get helicopters up in the air, you can get the firemen that you can find and get them to try and control it, mitigate risk, mitigate damage, get people evacuated, you do all these crazy things. And it’s such a big effort. At that point, it is you drop everything and you do that one thing. You just make sure that there’s no damage that’s out of control. Whereas there are ways that you can prevent fires.
And in fact, we’re having some trees taken out right now at our house for that exact reason because if there were to be a fire, these trees would be a danger and they would just burn uncontrollably and cause a lot of damage, not just to our house, but to the neighbor’s house and the entire neighborhood for that matter. So you can identify those areas that are about to tip over, so to speak, and make sure you address those problems before they get exacerbated.
Another analogy is you go to the doctor, you got your checkups, your annual checkups. It’s much easier to treat high blood sugar than once it becomes diabetes or something. I’m not a doctor, so I don’t know if that’s how it works, or you might have a weight problem and it’s easier to control that at the early stage before it gets to a heart problem, and then it’s a chronic disease at that point, and then you need medication and you’re at risk for all sorts of things. It’s the same kind of idea.

Julián Duque:
Yeah, it’s all about prevention through monitoring. What are those steps towards observability? As a Salesforce developer or as an admin, what do I need to take into account to start implementing observability in my org, in my applications?

Nikita Prokopev:
Great question. The answer to that is you can start very, very slow and very, very simple, and you can get into some crazy complexity.
So a lot of customers we speak to, they have very few resources. You’ve got your admin, maybe there’s a contractor or a couple more resources that are external, third party, and that’s it. And that’s all you can do. And they’re busy, they’ve got things to do, they’ve got businesses hounding them down, wanting to build stuff. But there are some easy things you can definitely start with.
So when we go to these events, the first step we tell everyone is like, “Look, you actually have observability in your org. It’s not readily accessible and easily consumable, so to speak, but it’s there and it’s your Apex Exception emails.” Salesforce alerts you to the fact that something is breaking, and there, you’ve got your flows, you’ve got your Apex code failures. In particular, the nasty ones that everyone hates, the governor limit breaches, like CBU timeouts, you got your heap size errors, you name it, one-on-one SOQL, everyone’s favorite. All these things come through email and more. You’ve got your email to case failures, a whole bunch of other things.
And part of it, I don’t think people pay enough attention to those, but there is a lot of good data in there. And it is data that’s telling you that you have a problem with the runtime aspect of your org. And just starting to pay attention to that alone will help you, maybe not take you to the next level, but it will at least alert you to the fact that, “Hey, I’m starting to see a lot of role locks, or I’m starting to see a lot of these flow failures, what’s going on?” So it can start pointing you in the right direction.
And then from there, you just got to explore your system. “Okay, if I have all these flow failures, let me go look at these flows. If I’m constantly hitting SOQL limits or heap size limits, let me go look at Apex.” You start identifying those problem areas and getting a plan to fix them. That’s the easiest possible thing you can do is to start paying attention to Apex Exceptions, making sure they go to an alias or a user that is aware of them and not go into the black void of nothingness, which a lot of these do go to. And it’s unfortunate. And then you’re completely blind.
Someone told me this anecdote. What is observability? And imagine driving 85 miles an hour on a highway and you close your eyes and keep driving. How long is it going to be before you crash?

Julián Duque:
Oh, no, no, no.

Nikita Prokopev:
It’s kind of scary, isn’t it?

Julián Duque:
That analogy gave me anxiety.

Nikita Prokopev:
I know. When I heard it for the first time, I was like, “Wow, that’s really vivid.” But that’s what happens today is you’re essentially going blind into this crazy world and your org’s running, your users are doing stuff, it’s completely out of your control, and you’re not even aware of what the problems might be. So that’s Apex Exceptions, fundamental and super easy.

Julián Duque:
Fundamental. Apex Exceptions, don’t ignore those. Read those emails.

Nikita Prokopev:
Please.

Julián Duque:
Or I don’t know, is there a way to automate those, maybe a catch-all or our way to read through them and sorting the ones that are important?

Nikita Prokopev:
There is. That’s where you start getting into some more advanced territory. And this is what we dedicate a lot of our talks to, trying to get people to think in that direction.
The easiest possible thing you can do, and when I talk about this, people kind of start looking around the room like, “Hey, what is this guy talking about? What do you mean?” But bear with me. It is doable. It’s been done. And what you could do is you could pipe them into your org, the easiest possible thing you could do. Salesforce has the tools to do that. You can build your email service and have those exceptions go to that email service.
And that’s where you get into… Well, in some sense, you’re shifting the problem from all these errors being in your inbox, but at the same time, you’re gaining a lot of advantages. So now, you’ve got things tracked in your organization, and let’s call them logs because that’s what they are. They’re essentially your execution logs, your exceptions. We’ll just call them logs for all practical purposes.
So you’re creating logs from those emails, but now you don’t just have the Outlook interface or the Gmail interface to start looking at them; you have the power of Salesforce and the platform. You’ve got, first of all, they’re tracked, so you’ve got history. Second of all, they’re out of your email, which in some companies, that could be considered a security vulnerability. You’ve got sensitive data in the logs, especially in screen flows where your data, some parts of your data may actually go into someone’s email. That’s not desirable for many companies. So you’ve got it tracked, and then you’ve got ways to now report, search, build dashboards, things like that on top of it.
Now, it’s still a far cry from an easy management of all of this complexity, but it gets you part of the way there. That’s step one.

Julián Duque:
At least you have visibility and a way for other users also to monitor what’s going on.

Nikita Prokopev:
Absolutely. I mean, just the historical perspective of it is it gets you a lot. Imagine seeing a graph of all your errors, and at some point, you see a spike. That right there, you’re not going to notice that in emails. But in the org, once you build a dashboard, it’s pretty obvious like, “Oh, today I had 150 errors. Oh my God, what happened today? Oh, some consultants pushed some code out,” and I’m not picking on consultants by the way. Maybe it’s an admin.

Julián Duque:
Yeah, you pick on them.

Nikita Prokopev:
He went and changed the flow that everybody used. He went and changed the flow that everybody uses or she or whoever. And now we have all of this, all these problems.
And by the way, I found that people are a little spooked by the idea of observability because all of a sudden, you can pin blame on somebody, and that’s not… I’ve always found that attitude to be strange. Like, okay, well let’s say you didn’t know. Then what would you do? Would you be better off or worse off? And the answer is, you’d be worse off because you’d be completely blind and the result would’ve been the same. You would’ve caused the problem. Now, you know why the problem’s there so you can make things better. The engineer in me never understood the idea of, “Okay, well if we don’t notice the problem, it’s not there.” It’s there. You just don’t know it’s there. So you’re better off knowing and iterating to make sure it doesn’t happen, and how can you iterate when you don’t know?
So anyway, I’m going in circles here, but that’s the point.

Julián Duque:
Great point. Well, and after logging those exceptions and errors, is there anything else I can do to at least improve my game before getting into a built solution, like the one you offer, for example?

Nikita Prokopev:
Yeah, absolutely. You can start… Well, this is something that some folks do today. You can get more granular with the logging. So the system notifications that Salesforce provides are great, and they notify you of problems. You can build on top of that log object using something like a logging framework where you can incorporate other things into your data collection of logs. You can start logging performance metrics. You can start doing some debugs. You can start doing UX metrics logging. You can enhance that error layer and add your own logging layer into the relevant parts of your logic. You can start doing that.
The next step would be to start making sense of this mess of logs. Start to organize them in some sort of way. And sometimes it’s as simple as, “Oh, I just want to group it by Apex class, or I want to group it by the flow that’s failing, or maybe I want to do something else.” If your org isn’t that complicated, you’ll find that it’s pretty easy to do so, to organize these errors.
And the error message is there, you can parse things out of it, whatever you like. They’re a fairly consistent format, so you can grab relevant parts of that error, start bringing them out to the top level of your log record, and then start grouping by that. And maybe you’ll start grouping them into issues. And these are your backlog items. So one user goes to a flow, breaks the flow, errors out. Second user does the same exact thing, breaks the same exact way. So you’ve got two logs, one issue. And then it’s going to start giving you more granular data around impact around the various problem areas. You could start getting into that realm.
Our product does something even more advanced where you get into fingerprinting. You could start looking at these error messages, grabbing different pieces from them, depending on what error it is, and then doing a hash over that. It gives you that kind of unique fingerprint and group it that way. That’s an even better way to do it. But it is far more complicated, especially if you want to make it very generic, suited for all purposes, but it’s doable.
And so once you’ve got control over your logs like that, when you group them into issues, you’ll find that you might have 2,000 logs overall, but you only have 20, 15 issues to deal with, and you start closing them out. From there, it’s just the sky’s the limit. You could start incorporating some project management aspects into it, start actioning these things, maybe time-to-cases or maybe integrate with Jira, start doing alerting of new issues or particularly bad ones. For example, one issue starts having more than X-amount of logs, so you could start doing things like that, build a Slack integration, nothing easier.

Julián Duque:
At least have something and start going one step at a time until you feel it’s the right amount.

Nikita Prokopev:
Exactly.

Julián Duque:
I know logging will give me, of course, way more visibility, but is there any performance impact while adding login to my code, for example? How much logging is too much?

Nikita Prokopev:
Yeah, great question. So there are a number of ways that logging can affect your performance, and one of those ways is unique to the platform because you’ve got limits to worry about.
So the first one and the most obvious one that most people like to pick on is, “Hey, I’m going to start logging all these things. I want to know everything. I want to see everything,” and you find that you run out of data storage in the org. I’m creating logs for every little thing. It becomes a problem because now your org is full, and so now you have to purge some of that data. So definitely be conscious of that. If you’re logging too much, you could get your org into unusable state.
But that said, these log objects, they tend to be fairly small, depending on what you want to capture. It’s still a handful of fields, so these records aren’t going to be too big. You’d have to start doing enormous amounts of logging. Most logging frameworks will have some sort of an idea of a log level where you could, even in debug logs, everyone knows in Salesforce, you’ve got your fine, finer, finest. You’ve got the various database and performance and all of these levels that you could start tweaking. So you can have a controller in your org that will… You can add all the logging you want, but then it will only save at the logging level that you specify. That’s another way to control that data volume problem.
Besides that, at the end of the day, depending on how you implement logging, you’re doing a DML, so that could be a problem in certain parts of your implementation. You’ve got to be conscious of callouts, you’ve got to be conscious of mixed DML. It’s not really a performance issue per se, but if you’ve got a complicated chain of triggers and automation, you could tip over the scales and blow through the limit if you’re doing too many DMLs like that.
What we found was it’s good to use platform events for that reason. There’s also another reason that you should use platform events, but platform events will kind of alleviate that burden a little bit. You don’t have to worry about callouts too much, DML statements and whatnot, but there are limits with platform events as well. So if you’re using them for complicated integrations, you might be running low on those allocations. You’ve got to be mindful of that too.
And at that point, you could start bulkifying things. Why do I need to save one log at a time? Let’s go and build a bunch of these, throw them in a collection, and then persist altogether. This is the philosophy behind Salesforce in general. Your triggers are bulkified. You’ve got to be conscious of that in your flows even. Best practice is you’ve got to populate your collection, do one update at the end. It’s the same idea.
So yeah, I think those are what comes to mind.

Julián Duque:
Wow. This is fascinating. I love it. This is a topic that I love a lot, and I used to work with observability tools for the Node JS ecosystem back in the day, so this topic is amazing.

Nikita Prokopev:
I’m glad you enjoy it. I do as well.

Julián Duque:
Yeah. When you are not building a observability tool or leading a company, what do you do in your free time?

Nikita Prokopev:
Oh, interesting. I do a lot of things. I have an eight-year-old daughter, so a lot of time is family time for us. We do all kinds of stuff. We go on hikes, we draw together. I’m also an artist, a weekend artist, so I’ve been doing fine art since probably 11 years old or something like that.

Julián Duque:
Oh, fine arts.

Nikita Prokopev:
It’s been a passion of mine, yeah. I like painting acrylics, pastels, pen and ink, that kind of thing. That’s a big hobby of mine.
I love music. I’ve been into, oh gosh, I’ve been into a lot of things over the years. My biggest musical influences are heavy metal and electronic music, and I play guitar. I’ve dabbled into production, which was a long time ago, but I really enjoyed it, and at some point I hope to pick it back up. Those are kind of my top hobbies, yeah.

Julián Duque:
That’s a lot. A man of many talents. Nikita, thank you. Thank you very much for joining us on the Salesforce Developer Podcast.

Nikita Prokopev:
Absolutely. Happy to be here.

Julián Duque:
Of course.
If you want to learn more about this show, head on to developer.salesforce.com/podcast where you can hear all the episodes and read the show notes. Thank you, everybody, and talk to you the next time.

 Get notified of new episodes with the new Salesforce Developers Slack app.