There are many open-source tools on the market. If we can follow them sufficiently, we can easily establish very nice big data systems. Is that all? So, if this is the case, why do many companies have very high costs for such services?
Most of the problem starts at this point. There is a big misperception in Turkey that if a software is open source, I can benefit from it in every way. I would like to include important details that are often overlooked here.
First of all, let’s consider what big data is. The term big data has started to be spoken/heard in almost every company and project, of course, when talking to people, everyone uses the same expression. 10K, 100K, 1M records are not measured as big data. If you have 1M user records, you can easily include them in select queries etc. with Postgresql. If you do good indexing, you will not even have problems with joins etc. I have seen very nice and high-performance joins in tables with 25M and 30M records. So what is this big data then?
Big data;
-
This happens at points where there is intense data production For example, you may need to make 100s or even 1000s of inserts in a minute. Of course, there are much more intensive systems. The important point here is that the system we use should be able to respond to reading requests without getting stuck while writing constantly. In Postgresql style relational db’s, writing operations performed as a continuous stream due to the transaction event will slow down the reading operations excessively.
-
Operations such as weighted aggregation or text mining are performed on the written data Since these operations do not comply with the working logic of relational db’s, we experience serious performance problems on relational db’s. For example, in a table with 1M records, a select query comes very quickly, and even if the indexes are correct, transaction management, sorting, pagination can be achieved with high performance in very low resources, but if you try to get sample monthly averages on the same dataset, you may be able to solve it a little with your data model, but it will not be very performant, and you also need to draw graphs according to months. If you say let’s get it as datasets, the db will be exhausted and maybe your resources will be completely exploited.
If you have one of these two parameters in your system, it can be said that you somehow need big data technologies. Of course, each vehicle that will be added to our ecosystem will return to us as a maintenance, update and monitoring burden, so this issue should be analyzed well and the choice should be made accordingly. In short, a tool is not used just to be used, you use it if you really need it.
Well, we analyzed our system and decided that we need such things called big data. What do we do?
When we start Big Data operations, all the relational db single instance and architectural designs used in classical programmer logic need to be overhauled. New concepts enter our lives. The first of these is the concept of reliable, which I think is the most important. This concept brings a reliable data source into our lives, apart from relational db nosql, so what does this mean? Think of a tool where you keep data. This tool does not give you a guarantee that the data is complete or up to date. You add data, you think that it has been written, but it cannot be added, because it just put it in the queue or ignored it for another reason. The workload in mint, which we call big data, is very high, and some issues are compromised in the tools prepared to handle this workload. The primary issue that is compromised is the security of the data. So what does it mean, like relational db, where did my one row record go? Or this record does not match the other record. Why was it inserted here? We shouldn’t ask such questions. There are different solutions to this, but I won’t go into detail about them. As we said at the beginning, we use it for purposes such as aggregate operations, analysis, graphics, etc. If you have more than 1M data, maybe 10s of them arrive every second, it is very possible that there may be some that we cannot receive. For example, some vehicles, if the RAM is too full, do not receive the incoming records at that moment in order to avoid crashing, or they put them in a temporary place and can add them 1 hour later, depending on the workload. As I said, we have to accept this issue, there are different alternatives in the solution.The other issue is modeling. There are many tools in the market as big data tools. Using these tools can be said to be logged in to bigdata somewhere, but the real issue is modelling. If you model the data you receive well, you can translate the most data with the least amount of resources. If we think about it this way, we have 1 GB of data and it took us 1 day to write it. I told the system to report this data to me in a different way outside of my model. This means that first convert this 1 GB data to the model I want and then respond to me with that model. Once the site receives this query, it will adapt it to the new model within 1 day and then it will be able to respond to you. It doesn’t make sense, right? Yes, many tools restrict you at the query stage, so you can’t query as you wish. That’s why you need to think about how you need the data beforehand and model it accordingly. So can I use too many models? The answer to this question depends on you. You have 1 GB of data. The vehicle you are using already keeps a few copies of this data in order to give you a quick answer (depending on your installation configuration settings). For example, it keeps 3 copies. It takes 3 GB of space. You kept it with 2 different models, and you will use 6 GB of space. This causes you to lose space and run your queries from two different points. If you design your model well instead, maybe you can solve this problem in a single model.
Yes, let’s come to the most important issue: choosing tools for Big data. This issue can be a problem in itself, I can almost hear you say, why do I use one of the many alternatives, they are all free anyway :). First of all, let’s clarify the concept of free. It is true that no one gives their sins to anyone for free, no one gives free, they definitely have an expectation in return. The important thing is to know this expectation and to identify the points where mutual interests match. Here comes the licensing issues, after all, open source does not mean that only the source code is open and you can use it as you wish. If you use it, you accept that license and you are expected to fulfill its requirements, otherwise you may be subject to legal sanctions. I’ve always wondered how the people who write these open source tools make money and why they write them, and here’s the answer. Let me explain briefly by grouping them without going into details;
In the 1st group, there are incomplete or faulty code shares that you will encounter a lot, the purpose here is for people to use these tools so that they cannot get out of trouble in critical areas :) and get support from us, or they earn money by sharing the missing version outside the version that works smoothly in their own ecosystem, such as Datastax, and saying that you cannot manage it, let me manage it for you. You choose your tools, you make all your development and design accordingly, you get stuck at a trick point, you cannot move forward, it is difficult to return the way you came, then your only option is to go and get support.
In the 2nd group, the SSPL license issue, which started recently with Mongo and Elasticsearch, is a problem in itself. Again, companies have launched the tools, many people have used them and built their products, now with this license, they are forcing everyone to open it until they reach your hardware design. Of course, since it is difficult for the companies using it to step back, they will either give money or open their designs, and the licensing companies will use the designs in people’s infrastructure. If you do not comply, you may face legal sanctions.
There are applications in the 3rd group that provide limited capabilities. If you do not buy the enterprise version, you will not be able to benefit from many features that you may need in the future. You will set up your architecture without being aware of this, and then when that day comes, it is the end of everything for you. For example, influx time series db open source gives you the opportunity to install a single instance. If you are going to install more than one instance, you need to purchase a license. In big data business, a single instance is not preferred in terms of speed and stability.Group 4 is the most beautiful group and it is my favorite group. Really, the aim is to develop it jointly and let us all benefit from it. For example, the chromium project was started and supported by Google, various improvements were made by the community and everyone, especially Google, Microsoft etc. benefited from the project. Companies take this community power behind them and support some projects in this way to develop joint projects. Projects such as Cri-o, Solr, Linux etc. are the products of this type of work. So what does this tool want? First of all, they say that you cannot sell this product and make money because the purpose is to serve the community. However, you can use these products downstream and earn money by selling the software you developed on them, or you can earn money by providing support, but you cannot directly sell this product and earn money.
These are just the things I wrote as a simple summary. For license details, you can review https://opensource.org/licenses here.
Going back to our article, we should pay attention to what we have decided to license when choosing a vehicle. First of all, we have to accept that there are still very few tools that we can call the best solution on the big data side, for example, Apache Lucene and Solr developed on it have proven their worth in the field, but at the same time, the companies that develop those tools and provide support have created models according to their own needs, so when you say time series db, you will see dozens of types, each of them is different from the other, you should pay attention to these features and choose accordingly. You will have to read a lot of tutorials in this part, but I do not like it at all :). So what do I do? I examine ecosystems and have to read tutorials again.
Of course, what I have written here is only a small part of the work. When choosing a tool, there are many factors such as stable operation, continued support from developers, etc.
NOTE: Do not use or let others use unlicensed products or products outside the scope of the license. We need to have a little empathy on this issue. My opinion is that using unlicensed software is THEFT.