Panel Discussion: Mono-repo Vs. Multi-repo Vs. Hybrid

And welcome, everybody, to our panel discussion. Today, we have a great topic. So mono repo versus hybrid versus multi-repo. And we have a lineup of speakers. So speakers, please go ahead, introduce yourself.

Luke. Hi, everyone. My name is Lukande, but a lot of people call me Luke. I'm a Star Wars fan so you can also call me Skywalker, if you like. And I'm from South Africa. My actual nationality is that I'm Zabian. Senior software engineer working at a company called Intellect Software in South Africa. And yeah, primarily in the financial services sector and focused mostly on the cloud and DevOps space.

Nice. And if you guys are a supporter of one of the approaches, please tell me which one. Luke, are you a supporter? Look, I'm between both multi and mono. But to be honest, most of the times I go with the mono approach, especially even my side projects. So, yeah, I guess I'd go with mono. OK, OK, Julie, hi. OK, found the unmute button. I'm still doing that after your COVID. But yeah.

So, hi, I'm Julie. I'm an engineer at Microsoft. I help customers use Azure on board to Azure. Before that, I was an enterprise architect, an insurance company, and helped a lot of developers figure out DevOps and mentored them, annoyed them, because I always have to do the security thing as well a little bit, be the bad person that says you have to sign your commits. Yeah. You're probably going to ask me now, mono, multi, hybrid. I've seen it all.

And welcome everybody to our panel discussion. Today we have a great topic. So, Monorepo vs. Hybrid vs. Multirepo and we have a line up of speakers. So, speakers please go ahead introduce yourself. Luke. Hi, everyone. My name is Lukande, but a lot of people call me Luke. I'm a Star Wars fan, so you can also call me Skywalker, if you like. And I'm from South Africa. My actual nationality is that I'm Zambian. I'm a senior software engineer working at a company called Entelect software in South Africa. And yeah, primarily in the financial services sector and focus mostly on the cloud and dev ops space.

Nice. And if you guys are a supporter of one of the approaches, please tell me which one, Luke. Are you a supporter? Look, I'm between both multi and mono. But to be honest, most of the time I go with the mono approach, especially even with my side projects. But I guess I'd go with mono. OK. OK. OK. Found the unmute button. I'm still doing that after your COVID. But yeah. So hi, I'm Julie. I'm an engineer at Microsoft. I help customers use Azure onboard to Azure. Before that, I was an enterprise architect, an insurance company and helped a lot of developers sort of figure out dev ops and mentored them, annoyed them because I always have to do the security thing as well, a little bit, sort of be the bad person that says you have to sign your commits. Yeah, you're probably going to ask me now mono, multi, hybrid.

OK. And I'm going to give the architects answer, which is, depends on the situation, which one's best. OK. Nice, nice. Victor, please go ahead.

Sure. So I'm Victor. I used to be at Google on the Angular team, and this is where I actually sort of not fell in love, but got to know monorepo development, what it can feel like. So Google, Facebook, Twitter, and others all use a very similar setup, right? And for the last couple of years, I've been leading a project called ANEX, which is a Google-style monorepo tool, for the masses. So if you don't have Google, but you want to do something similar, this is what it is. So obviously, I'm in the mono camp. But I actually don't care that much. You know, you can do whatever you want to do. OK, thank you. And Angel?

Yeah, so my name is Angel Rivera, developer advocate currently at CircleCI. And before that, I worked pretty much US military, working in Space Command, all the way through to small startups in Europe and the US, and then also went back to work for the US government for a little while. So I have a wide range of experience and a lot of things. To answer your question of which preferred repository, I'm a hybrid man. Wow. Definitely. Nice. Yeah.

OK, so the first question, could you please tell me and think about which top is better for different kinds of projects and why? Luke, please.

Sure. So I guess just from personal experience as well as learning from others, I think the monorepo approach is great when it comes to working with smaller sized projects or medium sized. And it's really good to help just with knowledge transfer and teams. That's one of the things that I like about it the most. You can easily have people integrated into getting a big picture of what the whole project is. They get to see if you have different services in that one repo.

I've seen it all. OK. And I'm going to give the architects answer, which is depends on the situation. Which one's best? OK. OK. Nice, nice. I like it.

Viktor, please go ahead. Sure. So I'm Viktor. I used to be at Google on a Angular team. And this is where I actually sort of, not fell in love, but sort of got to know monorepo style development, what it can feel like. So Google, Facebook, Twitter and others all use a very similar setup. And for the last couple of years, I've been leading a project called Annex, which is a Google style monorepo tool for sort of for the masses. So obviously, I'm in a Mono camp, but I actually don't care that much. You can do whatever you want to do. OK. Thank you. And Angelo?

Yeah. So my name is Angelo Rivera, Developer Advocate currently at CircleCI. And before that, I worked pretty much US military, working in Space Command all the way through to small startups in Europe and the US, and then also went back to work for the US government for a little while. So I have a wide range of experience in a lot of things. To answer your question of which preferred repository, I'm a hybrid, man. Wow! Definitely. Nice.

They get to see if you have different services in that one repo. They get to see the entire lay of the land, so to say. So yeah, that would be one of the major reasons why I think those would be good scenarios for a monorepo approach.

With multi-repo, I think if you're looking to have strong ownership, that's something that I think is becoming more and more popular, obviously, because of microservices. That's pretty good to have over there, in my opinion, if you want to have that kind of strong ownership and you want teams that are integrated to have that full end-to-end approach, then definitely. I am very keen to hear more about the hybrid approach and where there's room for that. So yeah, but those would be my thoughts.

OK, thank you. And Victor, what do you think about this? Yes, it's a good question in that it depends on how you define the monorepo. If you define a monorepo in the sense like the Google style monorepo or like the Facebook style, one super large repo, so obviously, that doesn't work for everyone. But on the other side, you have three projects side by side in a folder, and it's a monorepo as well. So if you'll think about the former, so very few companies should adopt the former because it requires a lot of energy expertise to just manage the stack. If you think about the latter, it works for any product, even if I'm building a blog, right? And the blog has two features like the list and the details. I can think of the list and the details being separate projects that my blog consists of, hence it's a monorepo, right? So it really depends, right? In terms of the rule of thumb that I use when I talk to companies who are helping to use monorepos is that if they do, basically, if they close it to a monolith, they have one large system, they can still do some structural development and partition to the packages, and treat it as a monorepo with multiple modules and be built independently and then assemble into a larger system. But the gain is not as big compared to when they have things like microfrontend, like microservices, because basically, what monorepos do well, I think, and that's what many companies realize, is that they help with coordination. Once you have a system consisting, let's say it consists of 20 nodes, right? I need to work on those 20 nodes in a structured way, share code, figure out how to deploy it. What is a system? Being able to spawn the 20 nodes in development and interact with it, right? That's where you need this extra tooling where the monorepo, one quote style tooling, that will help you to manage that system, right? And the more nodes you have, the bigger the gain is, right? So if you have 100 nodes, well, then you have to have something to help you manage 100 nodes, right? So obviously, it's sort of a chicken and egg thing. If you have good tooling, people create more nodes because it's easier, right? So it's kind of hard to gauge sometimes. But overall, I think counterintuitively, if you have microservices or microfrontends, you actually benefit more from monorepo because you need to coordinate them in some fashion in development and in deployment.

Okay, thank you. And Julia, what do you think about this? So I'm gonna lower my head. I feel old when I do stuff like that. So I've seen it all, right? Like I said before. And when I was an enterprise architect, I initially started actually as an engineer. And people talk about, oh, you should use this or that, but let's look at what it's like in real life, right? So we had something that was built. It was a distributed monolith and a monorepo, and how do you split that out? So it was okay, I think to start with a monorepo because we weren't sure what we were building, right? In terms of business domain, what do we actually need to do and execute in terms of features for this product to be successful for the user? So when you're kind of experimenting, you don't want to coordinate all that. Yes, you will shoot yourself in the foot in other places, running builds and whatnot, but in terms of just what is my product that the user actually sees, it's much easier than monorepo. So I think maybe after a year, we got to the point where we're like, oh, we got to take this apart now because also the engineers had developed enough skills that we really do want to separate out the single page application, and there was a multitier application, right? So there are different sort of layers that you have to go through, and again, this is enterprise. And so for a while, we had a distributed monolith. We split them out into repos, and then we pulled them in as Git sub modules, which was also less than ideal, but it's a little bit that sort of intermediary state until you get to the point, if you ever reached that point where you do have microservices, and not everybody reaches that point. Sometimes you go back and do a monorepo, and now as a, and I always have my architect hat on. I say, I don't care, just ship, whatever works for you.

Okay, Angel. Yeah, so I'm going to take a different approach to your question. I think that it really doesn't matter how you structure your code and where you structure it. I mean, these are just basically locations, right? Like where we're storing the stuff. And I think the problem that we all have is the tooling. And I think Victor has touched on that a little bit. The tooling is inefficient, right? I don't care where you work, Google, Facebook, CircleCI, Microsoft, the versioning control systems that we have are just not, they're not good. You've run them at scale, it's crap, right? We're not there yet, the industry. And you know, we've made huge progress, believe me. I've been around this business a long time, but the tooling is where the problems are because, right, if, I mean, Victor and, yeah, I think Victor was the one that worked at Google. They have a huge, right, monorepo, they had to build their own kind of internal tool to manage the versioning process, right? So let's start with the tooling. I think that's kind of where all the pain is coming from. And until that is bettered or evolved into a state where we can do versioning fairly proficiently, then I think some of these other problems will be irrelevant, right? Like that's why I said hybrid is kind of where I'm at because when you're talking repos and monorepos, it's literally just, where is that code being maintained and stored. And then the problem is how, the tooling that we use to version that stuff, right? So that's my answer.

Yeah, thank you so much. Yeah, you're right about the tooling. And let me ask them your short question and I supposed to hear like a short answer just about the tooling that you use. Like what is the preferable to top for you if you're talking about the monorepo, hybrid approaches, Julie? You can just please... Yeah, what is the preferable setup list of technologies that you use if you choose monorepo or let's say hybrid approach?

So here's the thing, I'm not sure I 100% agree with Angel. Why? Well, again, I have my enterprise architect hat on and the security vision issue, right? Who's working on something. You have one system, you always have that more management overhead to separate those people out. I come from compliant industries and that's always something you have to solve. And if you don't have enough people to do it, then you say, let's just split them up, all the different teams, even if they're building a distributive monolith.

Tooling, I don't know, I feel like tooling doesn't solve people problems. At the end of the day, a lot of this coordinating and that's what I mentioned in my talk, making sure everything goes well from your laptop to pushing to get to actually deploying. It's all a dance. It's all choreography more than the tooling. Give me anything, and yes, one will be better than the other, but what's really painful is working with other people. If I could just build my own thing and full stack, then no people, no problems. That's my answer. I'd push back on that.

Let's go. It's very obvious that we have a tooling problem. I mean, what you talked about, yes, I agree a hundred percent that people do get in each other's way. I think that's more of a DevOps kind of problem. So we're talking about culture at that point. What I'm talking about is repository indexing. Indexes becoming astronomically large, inefficient. That's the reason why Facebook, Twitter, Google all have their own internal in-house built system because of Git's inherent, slow indexing, extremely painful indexing. Once it gets to a point where you're talking about terabytes of data, actually even less than that in some cases.

So the other point to kind of counter that would be that yes, we have a people problem, but if you fix that inherent problem with the versioning tooling, you can also address the security problems that you're talking about. That's easily addressable through automation. I would throw CI-CD platforms at that security problem to collaborate around that. And we're seeing huge gains in security, right? Like even Azure, the GitHub now has bought Semmle, they acquired them, and now they have a slight, I would say, they do security scanning for you out the box on the platform.

I'm sure that's gonna grow into its own product line eventually, but right now you're getting it for free. So those security type questions are being answered currently. They're not perfect by any means, but I think with DevOps implementing that to handle the people portion, it is a separate issue than the version control systems and the reason why we have monorepo talks, or this conversation, my opinion.

I think we come from large industries, right? I'd love to hear what Luke, for example sees, because we're very skewed in the big Microsoft, Google's, and Facebook's, so I'd love to hear what Luke thinks. That's awesome. Luke? Yeah, so I'm in the red tape industry, man, with the financial. The shit. With the financial. Yeah, so he's not so small, right? Right.

And, look, one of the things that I find really challenging, and I would love to hear your opinions on this, is when you have huge teams working on a mono-repo, so they believe in the benefits that a mono-repo would provide, but they're trying to enforce that kind of system, and how does that work in terms of commit standards, and even standards in terms of best practices? So that's why I really do lean even towards, there's definitely a people element to this, and maybe starting from there. Tools, obviously, are an important factor, but I think I've seen a lot of the people problems, and that maybe if we started from there, and try and solve things from there, and build from there in terms of the paradigm we should be using, as opposed to, because not every project is alike, not every company is alike, and there's obviously gonna be a temptation with the bigger companies in the industry.

I've seen it where it's like, oh, but Google are practicing this, therefore, let's do that too. It was like, we're not Google. Our team is not structured like the teams they have there. We don't have the same kind of maturity they have in terms of the additional tooling they're using to make things more efficient. So yeah, those would be my thoughts on that particular area. Yeah, I just wanna clarify that, I mentioned teams, or the tooling piece being kind of the genesis of the problem, but I agree 100%. It's all about scale. So I'm talking about running these repos at scale, but yes, of course. And that's why I said hybrid, right? Like, remember that? Oh, yeah.

Victor, could you please share your thoughts? Sure, it's actually a very good, very interesting point, and I think that, like, it really depends, again, how you define a monorepo. If you look at the Google scale, or like Facebook scale, Microsoft scale, well, yeah, a lot of tools, for sure, do not work very well. You have to use like GVFS. A lot of interesting things, so you can actually, like, traverse the repo. So we, like, I personally work with a lot of large, I mean, large companies building large systems, but they don't have, like, one Uber monorepo that, you know, like, around the whole organization, right? It's usually like, we have 10 teams running a monorepo contribute to one system, right? Once you go to that scale, it's not that, you know, it works, right? Okay? So, like, you have to go past that, so I think that for most organizations, one Uber monorepo won't work anyhow for organizational reasons, right? It's like, you won't be able to, they can't even pay for the same, like, the same Azure box, right, to run CI, CD, and because two works have different budgets, right? So, like, let alone tooling, like, just, most organizations cannot agree just for business reasons, right, to move together. Right? So when I talk about monorepos and the benefits of those, I don't mean, like, Google monorepo scale-wise. I mean, Google monorepo style-wise, right? You have same type of tooling, but a smaller, a much, much, much smaller system, let's say, where 1,000 developers work only, right? Where you have maybe tens of millions of lines of code, right? We're not talking about a massive system that's around the world. More like, you know, we have ten apps that comprise this, you know, experience that most customers, perhaps, so, like, let's say, of a bank, right, interact with. Right? So at that point, tooling is less of an issue, right? When it comes to, like, how teams should coordinate, I think that it's true that, at the end of the day, everything ends up being, like, a people issue, and that, like, you have to work together, agree on standards and stuff, right? But it's not, like, people, like, I don't know which one goes first. So, like, if you, like, take some, like, social media as an example, right? And say, yes, if we were all normal people, it wouldn't be difficult in social media, it would be nice to each other, but we are not, right? But it's partially because social media forced us to be not nice to each other, right? Because of the way it, you know, the incentives that are there, right? Same with tooling.

If you have good tooling, it allows you to separate your code from everyone else's code, to enforce best practices in a relatively straightforward way, you know, like, people kind of coordinate better. Right, here, like, some of the pain goes away, and you have a nice experience, right, where you can actually deal with things.

So, I don't know which one is more primary, right? I think tooling, I'm a tools provider, so, obviously, I'm not an agile consultant, right? I'm providing tools, so, obviously, I'm skewed towards tooling, because I feel like that is something that you can, that has a relatively low cost, and can be integrated relatively fast, compared to change, like, change into, like, organizational structure, or culture, or even human nature, right? Which is a lot harder to do.

So, those things, okay, I, you know, I let ThoughtWorks people fight the battle, right? I'm not fighting that battle, I'm fighting the battle. Certain things should be simpler, right? Some interactions should be simpler. As such, the cost of an interaction is lower, right? And when it comes to a previous point, actually, I wanted to add a comment, but I wasn't sure if I should interject, and that, I actually think that another thing about Monorepo that I find interesting, that folks often assume that if you're on the same repo, you're necessarily more coupled to each other, right? But it's not hard to imagine the proper doing with the other way around, right? Google repo is much more coupled to each other than a Polar Repo organization, right? You can have boundaries within a Monorepo that are just as strict as if you had a Polar Repo. There are certain things you have to agree on, right, for sure, but it doesn't mean that you're necessarily building a convoluted monolith where you cannot draw organizational boundaries. You can draw organizational boundaries, you can do all the things that ConvOx's law would require you to do, right?

Okay, so it's not really a one thought because there are a lot of points made before, right? So just to tell you, like tooling is a foundational thing you can cheaply buy or integrate compared to lots of other things, right? And then once you have it, you need to solve people's issues because at the end of the day, that's what results in your code being cut loose and stuff. Sounds like you're in my camp with hybrid. I mean, I think the word hybrid as well. If you have a repo for a suborg, it is a monorepo for that org, right? If it does interact with other monorepos, like if you're on an island and you have a single boat going back and forth, it's not really a hybrid, it's still on your own, right? So I wouldn't call it a hybrid, but if that's how we define hybrid, then yes, right? I think that most organizations should go hybrid. Having one uber large repo for the organization, that just doesn't work, let alone tooling, right? They will be able to buy machines or whatever resources to pay for it because they won't be able to agree on budget, right? So there are lots of problems there for most orgs. Yeah, the hybrid for me is the ability to just, like I said, introduce the code bases you need to introduce to your automation, right? Your processes, your build processes, when you need to. So to me, it doesn't matter where these things live and how they live, if they're separate or if they're coupled I don't care, as long as my tooling can, you know, complete and execute tasks I need them to complete when I need them to complete them.

Yeah, no, I kind of get it. I haven't had a point that I know- I'm gonna interrupt because I've had my hand, virtual hand up for a while and now I have my physical hand. I just wanted to ask you, please.

Yeah. So let's backtrack everything a little bit, right? Like I just, I worry sometimes like, who is our audience? We're throwing so many things out there, right? Let's make this a little bit more concrete. We keep talking about tooling and getting stuff in repo. All right, so making it more concrete, what do we have for tooling? We have branch protection, we have pull requests. All right, how do you want to fit all that together? And when I talk to customers, I talk about, you know, business units, fruits and veggies. I have fruits and I have veggies, and then I have dev, QA, and production. Dev, QA, and production. Think about that math. It grows almost, it feels like exponentially. I'm terrible at maths, so probably not exponentially. But it is gonna be a combination of tooling and people. And just sit down and plan that, right? And it's your business driving how do you need to separate your groups. And use names, don't use food and bar.

Fruits and vegetables, how do they work together? And again, maybe I'm gonna ask for fellow panelists what other tools they think of, but the ones that always pop to my head first are pull requests and branch protection. Well, those elements of pull requests, right? Those are outside of the core versioning tools, right? So that's a product of GitHub. They call it something else in another product. So I think, looking at the tooling natively, and I agree 100%, there's a people, obviously element to that. but I think that it's a conversation to be had outside of the modern repo. Because once you figure out the continuous integration portions of your processes, that's the, in my opinion, the developer people portion, where they're collaborating around code, right? I don't think it matters where the code lives and how it gets integrated into your processes, right? So, for me, they're two separate things, though they are related.

Okay, thank you all. Next question about statistics. So, you just said a lot interesting about this, those approaches, but what about statistics? Could you share some statistics like development speed, again, CI-CD process, code duplications, and so on, indexing, and Joe. So, Luke, what do you think about this? Yeah, I guess I could speak a bit about the deployment bit, and just if I am understanding the question correctly. I mean, so you have, again, this is kind of going back to the whole it depends or case-by-case situation, right? So, with MonoRepo, depending on how you've structured it, you could have a relatively quick deployment stream in that regard, if it's structured right, again, and then depending on the additional tooling that you're using, and this is gonna vary. The norm of what you will find is that the larger that and more complex that that becomes, the slower things may get, but even that is still kind of subjective because it still comes down to, well, what tooling are you using? What is the paradigm that you're actually using in this approach, whereas naturally with a multi-repo because things are smaller and a more micro sort of basis, things will be a lot faster. So I think in that particular regard, you can expect things to move much faster, statistically speaking, but even then, it's so hard to say that that's a universal thing because this is very case by case, so to say, across projects. So yeah, those would be my thoughts on that.

And so, Angel, you already said about indexing security. What else you can mention here? Yeah, so you have to, I agree 100% with Luke and everyone else who mentioned this. This is case by case, right? Everyone's teams are different, and I think Julie even touched on this a little bit too, but you have to understand what... So to kind of go to Julie's point of bringing that back, we did kind of go all over the place and it's probably my fault, but I tend to do that. But to kind of refocus all of this, you gotta understand what you're inheriting, right? So like, as a developer, I've been to many organizations, I inherited people's code. We all inherit, right, at some point, some code. The biggest struggle that I ever had was exactly what we're talking about today is like, all right, what do I do with this, right? So it's a mono-repo. How do I, I come from the world of I'll decouple the hell out of things because I like smaller, more manageable bits, right? But you can't go into a company, like, I don't wanna say the name, but there's a customer of ours that's huge, huge company, been around for a while, they have a mono-repo that is immense, right? If they were to break that apart, it would just cost them millions, probably billions of dollars. We're not gonna do that. So we have to operate within our context. And that's my point, right, like first assess what's going on with your teams, your codebase, can things be decoupled, right? Slowly, maybe, but should you decouple them, right? Like, so if your monolith is functioning and it's awesome, great, I don't see why any reason why you should decouple it, but if there are improvements you can make that make things more efficient, to Luke's point, smaller codebases run faster, I'm all about efficiency, getting my builds durations to be as small as possible. So, look at things from that perspective, how can you optimize, but you also have that cost, right? Like, is it gonna cost the company or the organization millions of dollars to implement a change and what is that return gonna be on those changes, right? So that's why I, I'm a big, huge proponent of, I guess, the hybrid, because you do, you will have situations where you're gonna inherit these huge monolithic repositories and then you're also gonna have to integrate some sort of new technologies or new services with that mono, monolithic app, right? So do you couple your new microservices and merge them into that monolithic application or you just leave them decoupled and run in that hybrid state, right? Where you have this one project where it's just running in this load and then, or in this repo, where it's mono. And then you have these other, you know, smaller services that are maybe newly built and leave them as decoupled, right? I think the answer is leave them as hybrid, leave them decoupled if you think that that's the most optimal way. But if it's more efficient to, you know, couple them and have them in one, then do that as well. Like it has to be a choice made by the organization with their specific circumstances. I'll leave it at that.

Okay, I see. Thank you, Victor, would you like to share some statistics? So I think it's, if you want to make a case for poly or mono, you can always come up with some numbers to prove the case. So the way I, what I would measure, right? Like sort of the tooling performance and then the organization performance. Tooling performance would be things like CI and CD, average time, worst case time, things like that, right? And then, I mean, depending on how you measure, you can make mono poly or fast. For example, if you have a system consisting of like 40 plus four modules that can be in one repo mount, for repos they form like a diamond shape, yeah. So you can make a change to the bottom one. Do I care about verifying the bottom one? Or do I care about integrating the bottom one like across to the top one, right? And do I basically count this CI CD run in case of poly repo or all CI CD runs that will be required, right? Things like that. So if you start doing that, then, you know, depending on what you choose to do, right? I think it ends up being the same, right? In that, of course you can like everyone distributes CI, right, even if you have a small distribution distributed CI, you don't run into one agent. Once you start distributing CI, it doesn't really matter whether you have monorepo or polyrepo that you still run your CI on like 30 boxes at the same time or 50 boxes, how big your repo is, right? So you can always make your monorepo or polyrepo to be fast enough, whatever fast enough depends. To you, let's say it's like 20 minutes in worst-case scenario, right? That can be made full repo pretty much any size, right? Unless you have one large node can be, cannot be divided at which point again, it's a more point poly and monorepo because you know, you have the same node that can be divided in either case, right? So I think it doesn't really matter. So in terms of like tools performance can be made more with either, organization performs a different thing, right? Then you start measuring things like what if I want to share a code? Like, this is my intent as a developer. Like how long does it take for me to share a code between like myself and another team member or like a different, from a different project for example, right? Does it take an hour, a day, a week, a month, right? And in a monorepo, that is much smaller. That's the benefit of the monorepo, right? The sharing is easier, right? But if you measure different things, right, then you can make polyrepo look better. It really depends on what you care about, right? Most organizations I work with want to share more, not less. Right? There are two divided in the lines of business and stuff. Right? So for them, they look for sharing as something to optimize for, right? So the monorepo is good in that case, right? But if you have one glued together or can you want to divide them more, perhaps you want to share less, right? And for organizational reasons, at which point you could still use a monorepo to do that, but polyrepo gives you it sort of in a more natural way. Right? Yeah, we're almost running out of time. I'd like to ask Julie. Julie, this is your last answer. What do you think about this? It all depends. I think we can continue in the spatial chat for people who want to debate with us. Okay, I think. Thank you all. It was very interesting. Thank you. Bye-bye. All right, good luck. Have a good one. All right, bye-bye. Bye-bye.

Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye.

Nice. Yeah. Okay. So the first question, could you please tell me and think about which top is better for different kinds of projects and why? Luke, please. Sure. So I guess just from personal experience as well as learning from others, I think the monorepo approach is great when it comes to working with smaller-sized projects or medium-sized and it's really good to help just with knowledge transfer in teams. That's one of the things I like about it the most. You can easily have people integrated into getting a big picture of what the whole project is. They get to see if you have different services in that one repo, they get to see the entire lay of the land, so to say. That would be one of the major reasons why I think those would be good scenarios for a monorepo approach. With multi-repo, I think if you're looking to have strong ownership, that's something that I think is becoming more and more popular, obviously, because of microservices. That's pretty good to have over there, in my opinion. If you want to have that kind of strong ownership and you want teams that are integrated to have that full end-to-end approach, then definitely. I am very keen to hear more about the hybrid approach and where there's room for that. Those would be my thoughts. Thank you. Viktor, what do you think about this? Yeah, it's a good question, in that it depends on how you define the multi-repo. If you define the multi-repo in the sense of the Google-style multi-repo or whatever, like the Facebook-style, one super-large repo, obviously, that doesn't work for everyone. Right? And on the other side, you have three projects side-by-side in the folder, right? And it's a monitoring repo as well. Right? So if you think about the former... Okay, so very few companies should adopt the former, because it requires a lot of energy expertise to just manage the stack. Right? If you think about the latter, it works for any project, even if I'm building a blog, right? And the blog has two features, like the list and the details. I can think of the list and the details being separate projects that my blog consists of. Hence it's a monolith. Right? So it really depends, right? In terms of the rule of thumb that I use when I talk to companies who help to use monoliths is that if they do... Basically, if they close the tool to a monolith, at the app, like one large system, they can still do some structured development partition six of the packages, you know, and treat it as a monolith with multiple modules, and be built independently and then assembled into a larger system. But the gain is not as big. Right? Comparing to when they have things like microfrontend, the microservices, because basically, what monoliths do well, I think, and that's what many companies realize, is that they help with coordination. Right? Once you have a system, let's say it consists of 20 nodes. Right? I need to work on those 20 nodes in a structured way.

Right? Share code, you know, figure out how to deploy it. What is a system? Right? Being able to spawn the 20 nodes in development and interact with it. Right? That's where you need this extra tooling. Right? The monorepo, style tooling, that will help you to manage that system. Right? The more nodes you have, the bigger the gain is. Right? So if you have, you know, 100 nodes, well then you have to have something to help you manage 100 nodes. Right? So obviously, it's sort of a chicken and egg thing, if you have good tooling, you will create more nodes because it's easier. Right? So it's kind of hard to gauge sometimes. But overall, I think counterintuitively, if you have microservices and microfrontends, You actually benefit more from monorepo because you need to coordinate them in some fashion in development and in deployment.

Okay. Thank you. And Julie, what do you think about this? So I'm going to lower my head, I feel old when I do stuff like that. So I've seen it all, right? Like I said before. And when I was an enterprise architect, I initially started actually as an engineer. And people talk about, oh, you should use this or that. But let's look at what it's like in real life, right? So we had something that was built. It was a distributed monolith in a monorepo. And how do you split that out? So it was okay, I think, to start with a monorepo because we weren't sure what we were building, right? In terms of business, domain, what do we actually need to do and execute in terms of features for this product to be successful for the user.

So when you're kind of experimenting, you don't want to coordinate all that. Yes, you will shoot yourself in the foot in other places, running builds and whatnot. So, in terms of just what is my product that the user actually sees, it's much easier in the monorepo. So I think maybe after a year, we got to the point where we're like, oh, we got to take this apart now. Because also the engineers had developed enough skills that we really do want to separate out the single page application and there was a multitier application, right? So there are different sort of layers you have to go through. And again, this is enterprise. And so for a while, we had a distributed monolith. We split them out into repos and then we pulled them in as Git submodules, which was also less than ideal, but it's a little bit that sort of intermediary state until you get to the point, if you ever reach that point, where you do have microservices. And not everybody reaches that point. Sometimes you go back into a monorepo. And now, and I always have my architect hat on. I say, I don't care.

Just ship. Whatever works for you. Okay. Angel. Yeah. So I'm going to take a different approach to your question. I think that it really doesn't matter how you structure your code and where you structure it. I mean, these are just basically locations. Right? Like where we're storing the stuff.

And I think the problem that we all have is the tooling. I think Victor has touched on that a little bit. The tooling is inefficient. Right? So I don't care where you work. Google, Facebook, CircleCI, Microsoft. The versioning control systems that we have are just not… They're not good. It's true. You've run them at scale. It's crap. Right?

We're just not there yet. The industry. You know, we've made huge progress. Believe me. I've been around this business a long time, but the tooling is where the problems are. Victor was one that worked at Google. They have a huge monorepo. They had to build their own internal tool to manage the versioning process. Let's start with the tooling. I think that's where all the pain is coming from. And until that is bettered, or evolved into a state where we can do versioning fairly proficiently, then I think some of these other problems will be irrelevant.

That's why I said hybrid is where I'm at, because when you're talking repos and monorepos, where is that code being maintained and stored? And then the problem is the tooling that we use to version that stuff, right? So that's my answer. Thank you so much.

Yeah, you're right about the tooling. And let me ask you then a short question, I suppose to hear a short answer, just about the tooling that you use. What is the preferable setup for you, if you're talking about the monorepo, monorepo hybrid approaches, Julie?

Yeah, what is the preferable setup list of technologies that you use? If you choose monorepo or let's say hybrid approach? So here's the thing, I'm not sure I 100% agree with Angel. Why? Well, again, I have my enterprise architect hat on and the security issue, right? Who's working on something? When you have one system, you always have then more management overhead to separate those people out. I come from compliant industries and that's always something you have to solve.

And if you don't have enough people to do it, then you say, let's just split them up, all the different teams, even if they're building a distributed monolith. I don't know, I feel like tooling doesn't solve people problems. At the end of the day, a lot of this coordinating, and that's what I mentioned in my talk, making sure everything goes well from your laptop to pushing to Git to actually deploying, it's all a dance. It's all choreography more than the tooling. Give me anything and yes, one will be better than the other, but what's really painful is working with other people, right?

That's my answer. I'd be pushing back on that. Let's go. It's very obvious that we have a tooling problem. I mean what you talked about, yes, I agree a hundred percent that people do get in each other's way. I think that's more of a DevOps kind of problem, right? So we're talking about culture at that point.

What I'm talking about is repository indexing. Indexes becoming astronomically large, inefficient. That's the reason why Facebook, Twitter, Google all have their own internal in-house built system because of gets inherent slow indexing, extremely painful indexing. Once it gets to a point where you're talking about terabytes of data, actually even less than that, right? In some cases.

So the other point to kind of counter that would be that, yes, we have a people problem, if you fix that inherent problem with the versioning tooling, right? You can also address the security problems that you're talking about. That's easily addressable through automation. I would throw CI CD platforms at that security problem, right? To collaborate around that. And we're seeing huge gains in security, right? Like even Azure, the GitHub now has bought what was Semmle, they acquired them. And now they have a slight, I would say, they do security scanning for you out-of-the-box on the platform.

And now they have a slight, I would say, they do security scanning for you out-of-the-box on the platform. I'm sure that's going to grow into its own product line eventually, but right now you're getting it for free. So those security type questions are being answered currently. They're not perfect, by any means. But I think with DevOps implementing that to handle the people portion, it is a separate issue than the version control systems and the reason why we have monorepo talks, or this conversation. My opinion. I think we come from large industries, right? I'd love to hear what Luke, for example, sees, because we're very skewed in the big Microsoft, Google, and Facebook. So I'd love to hear what Luke thinks. That's us, can't Luke? Yeah. Yeah, so I'm in the red tape industry, man, with the financial... Yeah, so he's not so small, right? Right, and look, one of the things that I find really challenging, and I would love to hear your opinions on this, is when you have huge teams working on a monorepo, so they believe in the benefits that a monorepo would provide, but they're trying to enforce that kind of system. And how does that work in terms of commit standards? And even standards in terms of best practices. So that's why I really do lean even towards, there's definitely a people element to this, and maybe starting from there. Tools obviously are an important factor, but I think I've seen a lot of the people problems, and that maybe if we started from there, and try and solve things from there, and build from there in terms of the paradigm we should be using, as opposed to... Not every project is alike, not every company is alike, and there's obviously going to be a temptation with the bigger companies in the industry. I've seen it where it's like, oh, but Google are practicing this, therefore, let's do that too. It was like, we're not Google, and our team is not structured like the teams they have there. We don't have the same kind of maturity they have in terms of the additional tooling they're using to make things more efficient. So yeah, those would be my thoughts on that particular area. I just want to clarify that I mentioned teams or the tooling piece being the genesis of the problem, but I agree 100% it's all about scale. So I'm talking about running these repos at scale, but yes, of course. And that's why I said hybrid, right? Remember that. Oh, yeah. Victor, could you please share your thoughts? Sure. That's actually a very good, very interesting point. And I think that it really depends on, again, how you define a monorepo. If you look at a Google scale or a Facebook scale, Microsoft scale, well yeah, a lot of tools for sure do not work really well. You have to use like GVFS, do lots of interesting things so you can actually traverse your repo. So we, like, I personally work with a lot of large companies building large systems, but we don't have one uber-monorepo that runs the whole organization. It's usually like we have ten teams running a monorepo that contribute to one system.

And once you go to that scale, it's not that it works. So, you have to go past that. So I think that for most organizations, one uber-monorepo won't work anyhow, for organizational reasons. It's like you won't be able to… They can't even pay for the same Azure box to run CICD, because two works have different budgets. So, let alone tooling, like just those organizations cannot agree just for business reasons to move together.

When I talk about monorepos, and the benefits of those, I don't mean like Google monorepo scale-wise. I mean Google monorepo style-wise. You have the same type of tooling, but a smaller, a much, much, much smaller system. Let's say we have a thousand developers working on it, where you have maybe tens of millions of lines of code. We're not talking about a massive system that has run the world, more like we have ten apps that comprise this experience that most customers, perhaps, like, let's say, of a bank, interact with. So, at that point, tooling is less of an issue.

When it comes to how teams should coordinate, I think that it's true that, at the end of the day, everything ends up being a people issue, and that you have to work together, agree on standards and stuff. But it's not like people, like, I don't know which one goes first. Like, if you take social media as an example and say, yes, if we were all normal people, it wouldn't be difficult on social media. We'd be nice to each other. But we are not. But it's partially because social media forces us to be not nice to each other. Like, it's the incentives that are there. Same with tooling.

If you have good tooling that allows you to separate your code from everyone else's code, to enforce best practices in a relativistically forward way, people kind of coordinate better. Some of the pain goes away, and you have a nice experience where you can actually deal with things. So, I don't know which one is more primary. I think tooling... I'm a tools provider. So, obviously, I'm not an agile consultant. I'm providing tools. I'm more towards tooling because I feel like that is something that you can... They have a relatively low cost, and can be integrated relatively fast, compared to changing to, like, organizational structure, or culture, or even human nature, which is a lot harder to do.

So, those things... Okay, I, you know, I let ThoughtWorks people fight the battle. I'm not fighting that battle. I'm fighting the battle. Certain things should be simpler. Some interactions should be simpler, such that the cost of an interaction is lower, and when it comes to a previous point, actually, I wanted to add a comment, but I wasn't sure if I should interject in that. I actually think that another thing about monorepos that I find interesting, that folks often assume that if you're on the same repo, you're necessarily more coupled to each other, right? But it's not hard to imagine with Propa2 and BluBit, it doesn't work out, all right? Google repos look more coupled to each other than, like, a Polyrepo organization, right?

So there are some things you have to agree on, right, for sure, but it doesn't mean that you're necessarily building a convoluted monolith where you can draw organizational boundaries. You can draw organizational boundaries, you can do all the things that Conway's Law would require you to do, right? So it's not really a one thought because there are a lot of points made before, right? So just to clarify, tooling is a foundation of things that you can cheaply buy or integrate, compared to lots of other things, right? And then once you have it, you need to solve people's issues, because at the end of the day, that's what results in your code being kind of loose and stuff, right?

Sounds like you're in my camp with hybrid. I mean, I think the word hybrid as well, like if you have a repo for a suborg, it's more than a repo for that org, right? If it doesn't interact with other monolith, if you're on an island and you have a single boat going back and forth, it's not really a hybrid. You're still on your own, right? So I wouldn't call it a hybrid. But if that's how we define hybrid, then yes, right? I think that most organizations should go hybrid, having one over a large repo for the whole organization. That just doesn't work, let alone tooling, right? They won't be able to buy machines or whatever resources to pay for it because they won't be able to agree on budget, right? So like there's lots of problems there for most orgs.

The hybrid for me is the ability to just, like I said, introduce the code bases you need to introduce to your automation, right? Your processes, your build processes when you need to. So to me, it doesn't matter where these things live and how they live. If they're separate or if they're coupled, I don't care as long as my tooling can, you know, complete and execute tasks I need them to complete when I need them to complete them. Yeah. No, I kind of get it. I have another point that I know you want to say. I'm going to interrupt because I've had my virtual hand up for a while and now I have my physical hand. I just wanted to ask you, please.

I'm going to interrupt because I've had my virtual hand up for a while and now I have my physical hand. I just wanted to ask you, please. Yeah. So let's backtrack everything a little bit, right? Like I just, I worry sometimes like who is our audience? We're throwing so many things out there, right? Let's make this a little bit more concrete. We keep talking about tooling and getting stuff and repos, right? So making it more concrete, what do we have for tooling? We have branch protection, we have pull requests, right? How do you want to fit all that together? And when I talk to customers, I talk about, you know, business units, fruits and veggies. I have fruits and veggies, and then I have dev, QA and production, dev, QA and production, think about that math, it grows almost, it feels like exponentially. I'm terrible at maths, we're probably not exponentially. But it is going to be a combination of tooling and people. And just sit down and plan that, right? And it's your business driving how do you need to separate your groups. Use names, don't use food and bar, fruits and vegetables, how do they work together? And again, maybe I'm gonna ask a fellow panelist for other tools, they think of but the ones that always pop to my head first are pull requests and branch protection. Well, those elements of pull requests, right? Those are outside of, like, the core versioning tools, right? So that's a product of GitHub, you know, they call it something else in another product. So I think, you know, looking at the tooling natively, then I agree 100%, it's there's a people, obviously element to that. But I think that it's a conversation to be had outside of the modern repo, because once you figure out the continuous integration portions of your processes, right, that's the, in my opinion, the developer people portion where they're collaborating around code, right? I don't think it matters where the code lives and how it gets integrated into your processes. Right? So for me, they're two separate things, though they are related.

Okay, thank you all. Next question about statistics. So you just said a lot interesting about this, those approaches. But what about statistics? Could you share some statistics like development speed, again, CI, CD process, code applications and so on indexing, Angel. So Luke, what do you think about this? Yeah, I guess I could speak a bit about the deployment, bit and just if I am understanding correctly. I mean, so you have, again, this is kind of going back to the whole it depends or case by case situation, right? So with Monorepo, depending on how you've structured it, you could have a relatively quick deployment stream in that regard, if it's structured right again, and then depending on the additional tooling that you're using. And this is going to vary. The norm of what you will find is that the larger that and more complex that that becomes, the slower things may get. But even that is still kind of subjective, because it still comes down to, well, what tooling are you using? What is the paradigm that you're actually using in this approach? Whereas naturally, with a multi-repo, because things are smaller in a more micro sort of basis, things will be a lot faster. I think in that particular regard, you can expect things to move much faster, statistically speaking. But even then, it's so hard to say that that's a universal thing, because this is very case by case, so to say, across projects. So yeah, those would be my thoughts on that. Angel, you already said about indexing, security, what else you can mention here? Yeah. So you have to. I agree 100% with Luke and everyone else who mentioned this. This is case by case, right? Everyone's teams are different, and I think Julie even touched on this a little bit too.

But you have to understand, to kind of go to Julie's point of bringing that back. We did kind of go all over the place, and it's probably my fault, but I tend to do that. But to kind of refocus all of this, you got to understand what you're inheriting, right? As a developer, I've been to many organizations, I inherited people's code. We all inherit, right, at some point, some code. The biggest struggle that I ever had was exactly what we're talking about today, is, like, all right, what do I do with this, right? So, it's a monorepo, how do I... I come from the world of, I'll decouple the hell out of things, because I like smaller, more manageable bits, right? But you can't go into a company like, I don't want to say the name, but there's a customer of ours that's huge, huge company, been around for a while, they have a monorepo that is immense, right? If they were to break that apart, it would just cost them millions, probably billions of dollars. We're not going to do that, so we have to operate within our context. And that's my point, right? Like, first assess what's going on with your teams, your code base, can things be decoupled, right? Slowly, maybe, but should you decouple them, right? Like, so if your monolith is functioning and it's awesome, great, I don't see any reason why you should decouple it. But if there are improvements you can make that make things more efficient, to Luke's point, smaller code bases run faster. I'm all about efficiency, getting my builds' durations to be as small as possible. So, look at things from that perspective. How can you optimize? But you also have that cost, right? Like, is it going to cost the company or the organization millions of dollars to implement a change? And what is that return going to be on those changes? So, that's why I'm a big, huge proponent of, I guess, the hybrid, because you will have situations where you're going to inherit these huge monolithic repositories, and then you're also going to have to integrate some sort of new technologies or new services with that monolithic app, right? So, do you couple your new microservices and merge them into that monolithic application, or you just leave them decoupled and run in that hybrid state, right? Where you have this one project where it's just running in this load, and then in this repo, where it's mono, and then you have these other, you know, smaller services that are maybe newly built, and leave them as decoupled, right? I think the answer is leave them as hybrid, leave them decoupled, if you think that that's the most optimal way. But if it's more efficient to, you know, couple them and have them in one, then do that as well. Like, it has to be a choice made by the organization with their specific circumstances. I'll leave it at that.

Okay, I see. Thank you. Victor, would you like to share some statistics? Sure, so I think it's… If you want to make a case for Polyamore, you can always come up with some numbers to prove the case. So the way I… what I would measure, right? Like the tooling performance and then the organization performance. Tooling performance would be things like CICD, average time, worst case time, things like that, right? And then, I mean, depending on how you measure, you can make more than one repo faster. For example, if you have a system consisting of 40 repos or 4 modules that can be in one repo, multiple repos, they form a, let's say, a diamond shape, yeah? They will make a change to the bottom one. Do I care about verifying the bottom one or do I care about integrating the bottom one like across to the top one, right? And do I basically count this CICD run in case of poly-repo or all CICD runs? That will be required, right? Things like that. So, if you start doing that, then, you know, depending on what you choose to do, right? I think it ends up being the same, right? And that, of course, you can, like, everyone distributes CI, right? Even if you have a small repo that you distribute CI, you don't run on one agent. Once you start distributing CI, it doesn't really matter if you have monorepo or poly-repo in that you still run your CI on like 30 boxes at the same time, or 50 boxes, however big your repo is, right? So, you can always make your monorepo or poly-repo to be fast enough. Whatever fast enough depends... Do you want to say it's like 20 minutes in worst-case scenario, right? That can be made for a repo pretty much any size, right? Unless you have one large node that cannot be divided, at which point, again, it's a moot point between poly- and monorepo, because you have the same node that can be divided in either case, right? So, I think it doesn't really matter. So, in terms of, like, tools performance can be made work with either. Organization performance is a different thing, right? Then you start measuring things like, what if I want to share a code, like this is my intent as a developer, like, how long does it take for me to share a code between, like, myself and another team member, or like, from a different project, for example, right? Does it take an hour, a day, a week, a month, right? And in a monorepo, that is much smaller. That's the benefit of the monorepo, right? The sharing is easier, right? But if you measure different things, then you can make poly-repo look better. It really depends on what you care about, right? Most organizations I work with want to share more, not less, right? They are too divided into lines of business and stuff, right, so for them, they look for sharing as something to optimize for, right? So the monorepo is good in that case, right? But if you have one, you know, glued together, or can you want to divide them more, perhaps you want to share less, right? For organizational reasons, at which point, you could still use a monorepo to do that, but polyrepo gives you a sort of, in a more natural way, right?

Yeah, we're almost running out of time. I'd like to ask Julie, Julie, that's your last answer. What do you think about this? It all depends. I think we can continue in the spatial chat for people who want to debate with us. Okay, I think. Thank you all. That was very interesting. Thank you. Bye-bye.