Choosing Data Processing Architecture: Key Questions to Consider

Choosing Data Processing Architecture: Key Questions to Consider

Wojciech Marusarz - July 14, 2023

As businesses grapple with increasing volumes of data and a pressing need for real-time insights, choosing the right data processing architecture becomes critical. In particular, deciding between batch processing, stream processing, and hybrid models such as the Kappa and Lambda architectures can be challenging.

Before making this crucial decision, it’s important to ask a series of questions to ensure your selected architecture fits your organization’s unique needs. In this article, you will find ten key considerations.


The nature and volume of your data are fundamental factors in selecting your data processing architecture. The kind of data you deal with, be it structured or unstructured, and the scale of data you’re working with significantly influence your choice.

Real-time processing or batch processing? The answer depends on your required processing speed. Real-time insights call for stream processing, while less time-sensitive tasks can be handled through batch processing.

Tolerance for latency is closely tied to the processing speed and could determine whether stream or batch processing is a better fit.

The consistency requirements of your system can help choose between Lambda and Kappa architectures. If your application can tolerate eventual consistency, Lambda architecture could be suitable. For applications requiring immediate consistency, consider Kappa architecture.

Fault tolerance needs are essential, especially if your system cannot afford data loss. Both Kappa and Lambda architectures offer robust fault tolerance.

As your data grows with your organization, the scalability of your architecture becomes a crucial consideration.

The complexity of computations often necessitates batch processing, while simpler, quick computations are better suited to stream processing. Though, if you need to run complex computations, like machine learning models, over real time data streams, you will still require a dedicated architecture to accommodate those needs.

Your storage requirements guide your choice of architecture, with considerations around whether data needs to be stored for lengthy periods or be available for random access.

Budget considerations are pivotal, as different architectures come with varying costs associated with setup, maintenance, and scaling.

Lastly, your team’s expertise will significantly impact your choice of architecture. You may need to consider additional training, new hires, or leveraging external services.

Implementing modern big data architectures can be complex and requires deep expertise. If you’re unsure or need assistance optimizing your current setup, nexocode’s data engineering experts are here to help. We can guide you through the process, ensuring a robust, scalable, and cost-effective solution for your unique needs. Contact us today to take the first step towards a tailored, effective data strategy.

In the era of digital transformation, businesses generate vast amounts of data at an astonishing pace. This data, ranging from customer interactions and transaction logs to IoT sensor readings, holds immense potential. Extracting insights from this data allows businesses to enhance their decision-making, improve operations, and create innovative products and services. However, to unlock this potential, the data must be efficiently managed and processed – and this is where data processing architecture comes into play.

Data processing architecture refers to the system that handles the organization, processing, and analysis of data. It encompasses the methods, techniques, and technologies used to collect, ingest, process, analyze, and store data. The choice of a suitable architecture is critical as it governs how swiftly and effectively a system can turn raw data into valuable insights.

Choosing the right data processing architecture is a strategic decision that influences the speed and efficiency of data-driven insights. It is critical to consider factors such as data volume, processing speed requirements, fault tolerance, scalability, and budget when deciding on an architecture. With the appropriate architecture in place, businesses can harness their data’s full potential and transform it into meaningful, actionable insights.

Batch vs. Stream Processing

When choosing a data processing architecture, one of the key decisions is whether to use batch processing, stream processing, or a combination of both. Both batch and stream processing have their own unique strengths and are suited to different kinds of tasks and business needs.

Batch Processing

Batch Processing involves collecting data over a period of time and processing it in large “batches”. This method is particularly effective when dealing with vast quantities of data and where immediacy is not a concern. Since data is processed in large batches, computational resources are utilized efficiently, and the overhead involved in initiating and terminating tasks is minimized. This makes batch processing an ideal choice for tasks such as daily reports, data synchronization, and backup operations.

However, batch processing is not suitable for applications that require real-time insights or immediate response. The latency involved in batch processing - from the time data is collected to the time it’s processed and results are available - can range from a few minutes to several hours, depending on the size of the batch and processing complexity.

How does batch processing work?

How does batch processing work?

Stream Processing

Stream Processing, on the other hand, involves processing data in real-time as it arrives. This is ideal for use-cases that require immediate action based on the incoming data, such as real-time fraud detection, live analytics, and event monitoring. Stream processing can deliver insights with minimal latency, often in milliseconds, allowing businesses to respond to events as they occur.

Data stream processing

Data stream processing

However, stream processing can be computationally intensive and may require robust infrastructure, especially when dealing with high data velocity. Additionally, complex computations or analytics that need a holistic view of the data may not be suitable for stream processing.

Continuous stream processing - stream processing tools run operations on streaming data to enable real time analytics

Continuous stream processing - stream processing tools run operations on streaming data to enable real time analytics

Lambda vs. Kappa Architecture

When it comes to implementation models organizations usually need to combine elements of batch and stream processing. Data processing architectures that are available here are Lambda and Kappa models.

Lambda Architecture

Lambda Architecture is a data processing architecture designed to handle massive quantities of data by taking advantage of working on data streams to support both batch and stream processing methods. It divides the data processing into two paths - a batch layer that provides comprehensive and accurate views, and a speed layer that compensates for the latency of the batch layer by providing real-time views. This architecture provides a balance between the efficiency of batch processing and the immediacy of stream processing. However, it can be complex to maintain because it requires running, debugging, and maintaining two separate systems.

Real-time data processing with speed layer and separate batch layer in Lambda Architecture

Real-time data processing with speed layer and separate batch layer in Lambda Architecture

Kappa Architecture

Kappa Architecture, in contrast, is a simplification of the Lambda Architecture. Instead of maintaining two separate paths for data processing, Kappa Architecture uses a single path - a single stream processing engine. It treats all data as a stream, thereby reducing the system’s complexity. This architecture can provide real-time insights with less maintenance overhead than the Lambda architecture. Historical data is still processed and stored as streams with bounded data streams contexts. However, it requires that all processing can be done effectively in a streaming manner, which may not be feasible for all types of computations or analytics.

Real-time data processing in Kappa Architecture

Real-time data processing in Kappa Architecture

Choosing between batch vs. stream processing or Lambda vs. Kappa architecture depends largely on the specific needs of your use case, including factors like data volume, processing speed requirements, tolerance for latency, and system complexity.

A comparison between traditional batch architectures and modern stream processing infrastructure that implements kappa pattern

A comparison between traditional batch architectures and modern stream processing infrastructure that implements kappa pattern

Questions to Ask When Deciding on Data Processing Architectures

When navigating the process of choosing the right data processing architecture for your business needs, it’s crucial to address a set of fundamental questions. This decision-making process is vital for developing data systems that can effectively handle and process your organization’s data – structured and unstructured data alike. Let’s delve into these crucial questions that help in deciding between two data processing architectures, namely batch processing and stream processing, and hybrid models such as Lambda and Kappa architectures.

1. Nature and Volume of Data

What is the nature and volume of data? Understanding the type of data (structured, unstructured, semi-structured) and the volume of data you’re dealing with can greatly influence your choice of architecture.

One of the principal determinants of your data processing architecture is the nature and volume of your data. Are you dealing with structured or unstructured data? The type of data influences your choice of architecture. For instance, unstructured data may be best suited for a data lake approach due to its flexibility. Also, the sheer volume of data you’re working with is significant. Vast data volumes might call for an architecture designed for high-throughput processing like stream processing or a distributed batch processing system.

2. Required Processing Speed

What is the required processing speed? Consider whether your use case demands real-time, near-real-time, or batch processing. This can help you determine if you need stream processing, batch processing, or a combination of both.

Understanding the speed at which you need to process data is essential when selecting a data processing architecture. If you need real-time data processing, stream processing can be more suitable, allowing you to handle new data streams as they come in. Conversely, if your processing can happen periodically or isn’t time-sensitive, batch processing, which processes data in batch cycles, could be a better fit.

3. Tolerance for Latency

What is your tolerance for latency? The importance of low-latency results may guide the decision between stream and batch processing.

Closely tied to the processing speed is your system’s tolerance for latency. Applications that require low-latency results, such as real-time fraud detection or event data monitoring, might be most appropriate for a stream processing system. On the other hand, batch processing can be utilized for tasks where latency is less of a concern.

4. Consistency Requirements

What are the consistency requirements? Some systems might need stronger consistency guarantees than others. Does your use case require immediate consistency, or can eventual consistency be tolerated?

Depending on the nature of your system, you might need stronger consistency guarantees. If your application can tolerate eventual consistency - delays between updates and accessing the updated data, the Lambda architecture, which operates a batch layer and speed layer simultaneously, might be suitable. However, the Kappa architecture, which relies on a single stream processing engine, would be preferred for applications requiring immediate consistency.

5. Fault Tolerance Needs

What are the fault tolerance needs? If your system cannot afford to lose any data due to a failure, you will need a robust architecture that includes failover and redundancy features.

System reliability is another key factor to consider. If your system cannot afford to lose any data due to a failure, you’ll need a robust, fault-tolerant architecture. Both Kappa and Lambda architectures offer strong fault tolerance, but their implementation can impact their effectiveness.

6. Scalability

What level of scalability do you need? If your data volume is expected to grow significantly over time, you need an architecture that can scale with your data.

As your organization grows, so too will your data. You must select a processing architecture that can scale with your data. Stream processing architectures like Kappa are designed to handle data in real-time and are, therefore, inherently scalable. However, well-designed batch processing systems can also scale effectively with the growth of existing data.

7. Complexity of Computations

What is the complexity of the computations? Complex computations might be more suitable for batch processing, while simple computations that need to be done quickly might be better suited for stream processing.

The complexity of the computations you need to perform can also influence your choice of architecture. Complex computations often necessitate batch processing, especially when dealing with intricate machine learning models requiring complete data set access. These models often need to sift through vast amounts of historical data to make accurate predictions, which is where batch processing can provide the most value.

On the other hand, simple, quick computations, or lightweight machine learning models that need real-time input and deliver immediate results, can be better suited to stream processing platforms. Stream processing is ideal for models where immediacy is paramount and the model’s effectiveness is not heavily dependent on large volumes of existing data (e.g., fraud detection models, anomaly detection, instant AI-based optimization models, etc.).

8. Storage Requirements

What are your storage requirements? If your data must be stored for a long period or must be available for random access, you need an architecture that can handle these storage requirements.

Storage requirements play a significant role in shaping your data processing architecture. Different storage solutions may be more beneficial depending on the nature of your data and how it will be used. You might require a robust data storage system if you have vast amounts of structured and unstructured data that need to be stored for extended periods or be available for random access.

A data architecture like Lambda could be fitting, as it provides both batch and speed layers for comprehensive data management. It ensures data quality by dealing effectively with raw data, historical data, and incoming data streams. On the other hand, if your focus is on processing data on the flow and then storing structured, processed data efficiently, a data mesh based on Kappa architecture could be more beneficial.

9. Budget

What is your budget? Different architectures may come with different setup, maintenance, and operation costs. Consider the financial resources available.

Budget considerations are pivotal when choosing a data processing architecture. Different architectures come with varying costs associated with setup, maintenance, and scaling. For instance, implementing a real-time data processing system such as the Kappa architecture might involve significant initial setup costs. However, these initial costs may be offset by lower long-term costs due to Kappa’s simpler, single-path processing model, making it a cost-effective choice for specific use cases.

Whether you are building a big data architecture from scratch or upgrading your existing data systems, the decision should align with your financial resources and long-term vision.

10. Team’s Expertise

What is your team’s expertise? When choosing an architecture, it’s important to consider your team’s skills and experience. Some architectures may require knowledge or skills your team does not have.

Lastly, your team’s expertise is a decisive factor in choosing the right data processing architecture. Implementing and maintaining complex data systems require specific skill sets. For example, deploying a big data architecture like Lambda or developing a stream processing system might require a deep understanding of data flows, machine learning, and various data processing platforms.

It’s essential to consider whether your team is equipped with these skills or if there’s a need for additional training or new hires. If the required expertise is not available in-house, you might want to consider engaging data strategy consulting services. These experts can provide invaluable insights and guidance on the most suitable data processing architecture based on your specific needs.

In some cases, your organization might also benefit from outsourcing specific tasks to data engineering services. These professionals can help design, build, and manage robust data systems, freeing up your internal team to focus on core business functions.

Closing Remarks: Making the Right Decision in Data Processing Architecture

Choosing the right data processing architecture is paramount and should align with your business needs, available resources, and long-term goals. It forms the backbone of your data strategy, and its effectiveness determines how well your organization can transform raw data into actionable insights.

Remember, building a data-driven culture within your organization is just as important as the technical aspects of data processing. A data-driven culture promotes informed decision-making, innovation, and a proactive approach to problem-solving. It sets the stage for your organization to leverage data as a strategic asset.

At the same time, recognize that the complexity of modern big data architecture necessitates expertise in various areas - from understanding the nuances of batch and stream processing to managing complex data flows and deploying machine learning models.

If you’re unsure where to start or need help optimizing your current setup, don’t hesitate to seek expert assistance. At nexocode, our data engineering experts are well-versed in helping organizations implement modern big data architectures tailored to their unique needs. We can guide you through the process, ensuring that you have a robust, scalable, and cost-effective solution that empowers your business to make the most of your data.

Take the first step towards enhancing your data strategy and fostering a data-driven culture. Contact us today to learn more about how nexocode can help transform your data processing architecture.

About the author

Wojciech Marusarz

Wojciech Marusarz

Software Engineer

Linkedin profile Twitter Github profile

Wojciech enjoys working with small teams where the quality of the code and the project's direction are essential. In the long run, this allows him to have a broad understanding of the subject, develop personally and look for challenges. He deals with programming in Java and Kotlin. Additionally, Wojciech is interested in Big Data tools, making him a perfect candidate for various Data-Intensive Application implementations.

Would you like to discuss AI opportunities in your business?

Let us know and Dorota will arrange a call with our experts.

Dorota Owczarek
Dorota Owczarek
AI Product Lead

Thanks for the message!

We'll do our best to get back to you
as soon as possible.

This article is a part of

Becoming AI Driven
92 articles

Becoming AI Driven

Artificial Intelligence solutions are becoming the next competitive edge for many companies within various industries. How do you know if your company should invest time into emerging tech? How to discover and benefit from AI opportunities? How to run AI projects?

Follow our article series to learn how to get on a path towards AI adoption. Join us as we explore the benefits and challenges that come with AI implementation and guide business leaders in creating AI-based companies.

check it out

Becoming AI Driven

Insights on practical AI applications just one click away

Sign up for our newsletter and don't miss out on the latest insights, trends and innovations from this sector.


Thanks for joining the newsletter

Check your inbox for the confirmation email & enjoy the read!

This site uses cookies for analytical purposes.

Accept Privacy Policy

In the interests of your safety and to implement the principle of lawful, reliable and transparent processing of your personal data when using our services, we developed this document called the Privacy Policy. This document regulates the processing and protection of Users’ personal data in connection with their use of the Website and has been prepared by Nexocode.

To ensure the protection of Users' personal data, Nexocode applies appropriate organizational and technical solutions to prevent privacy breaches. Nexocode implements measures to ensure security at the level which ensures compliance with applicable Polish and European laws such as:

  1. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (published in the Official Journal of the European Union L 119, p 1); Act of 10 May 2018 on personal data protection (published in the Journal of Laws of 2018, item 1000);
  2. Act of 18 July 2002 on providing services by electronic means;
  3. Telecommunications Law of 16 July 2004.

The Website is secured by the SSL protocol, which provides secure data transmission on the Internet.

1. Definitions

  1. User – a person that uses the Website, i.e. a natural person with full legal capacity, a legal person, or an organizational unit which is not a legal person to which specific provisions grant legal capacity.
  2. Nexocode – NEXOCODE sp. z o.o. with its registered office in Kraków, ul. Wadowicka 7, 30-347 Kraków, entered into the Register of Entrepreneurs of the National Court Register kept by the District Court for Kraków-Śródmieście in Kraków, 11th Commercial Department of the National Court Register, under the KRS number: 0000686992, NIP: 6762533324.
  3. Website – website run by Nexocode, at the URL: whose content is available to authorized persons.
  4. Cookies – small files saved by the server on the User's computer, which the server can read when when the website is accessed from the computer.
  5. SSL protocol – a special standard for transmitting data on the Internet which unlike ordinary methods of data transmission encrypts data transmission.
  6. System log – the information that the User's computer transmits to the server which may contain various data (e.g. the user’s IP number), allowing to determine the approximate location where the connection came from.
  7. IP address – individual number which is usually assigned to every computer connected to the Internet. The IP number can be permanently associated with the computer (static) or assigned to a given connection (dynamic).
  8. GDPR – Regulation 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of individuals regarding the processing of personal data and onthe free transmission of such data, repealing Directive 95/46 / EC (General Data Protection Regulation).
  9. Personal data – information about an identified or identifiable natural person ("data subject"). An identifiable natural person is a person who can be directly or indirectly identified, in particular on the basis of identifiers such as name, identification number, location data, online identifiers or one or more specific factors determining the physical, physiological, genetic, mental, economic, cultural or social identity of a natural person.
  10. Processing – any operations performed on personal data, such as collecting, recording, storing, developing, modifying, sharing, and deleting, especially when performed in IT systems.

2. Cookies

The Website is secured by the SSL protocol, which provides secure data transmission on the Internet. The Website, in accordance with art. 173 of the Telecommunications Act of 16 July 2004 of the Republic of Poland, uses Cookies, i.e. data, in particular text files, stored on the User's end device.
Cookies are used to:

  1. improve user experience and facilitate navigation on the site;
  2. help to identify returning Users who access the website using the device on which Cookies were saved;
  3. creating statistics which help to understand how the Users use websites, which allows to improve their structure and content;
  4. adjusting the content of the Website pages to specific User’s preferences and optimizing the websites website experience to the each User's individual needs.

Cookies usually contain the name of the website from which they originate, their storage time on the end device and a unique number. On our Website, we use the following types of Cookies:

  • "Session" – cookie files stored on the User's end device until the Uses logs out, leaves the website or turns off the web browser;
  • "Persistent" – cookie files stored on the User's end device for the time specified in the Cookie file parameters or until they are deleted by the User;
  • "Performance" – cookies used specifically for gathering data on how visitors use a website to measure the performance of a website;
  • "Strictly necessary" – essential for browsing the website and using its features, such as accessing secure areas of the site;
  • "Functional" – cookies enabling remembering the settings selected by the User and personalizing the User interface;
  • "First-party" – cookies stored by the Website;
  • "Third-party" – cookies derived from a website other than the Website;
  • "Facebook cookies" – You should read Facebook cookies policy:
  • "Other Google cookies" – Refer to Google cookie policy:

3. How System Logs work on the Website

User's activity on the Website, including the User’s Personal Data, is recorded in System Logs. The information collected in the Logs is processed primarily for purposes related to the provision of services, i.e. for the purposes of:

  • analytics – to improve the quality of services provided by us as part of the Website and adapt its functionalities to the needs of the Users. The legal basis for processing in this case is the legitimate interest of Nexocode consisting in analyzing Users' activities and their preferences;
  • fraud detection, identification and countering threats to stability and correct operation of the Website.

4. Cookie mechanism on the Website

Our site uses basic cookies that facilitate the use of its resources. Cookies contain useful information and are stored on the User's computer – our server can read them when connecting to this computer again. Most web browsers allow cookies to be stored on the User's end device by default. Each User can change their Cookie settings in the web browser settings menu: Google ChromeOpen the menu (click the three-dot icon in the upper right corner), Settings > Advanced. In the "Privacy and security" section, click the Content Settings button. In the "Cookies and site date" section you can change the following Cookie settings:

  • Deleting cookies,
  • Blocking cookies by default,
  • Default permission for cookies,
  • Saving Cookies and website data by default and clearing them when the browser is closed,
  • Specifying exceptions for Cookies for specific websites or domains

Internet Explorer 6.0 and 7.0
From the browser menu (upper right corner): Tools > Internet Options > Privacy, click the Sites button. Use the slider to set the desired level, confirm the change with the OK button.

Mozilla Firefox
browser menu: Tools > Options > Privacy and security. Activate the “Custom” field. From there, you can check a relevant field to decide whether or not to accept cookies.

Open the browser’s settings menu: Go to the Advanced section > Site Settings > Cookies and site data. From there, adjust the setting: Allow sites to save and read cookie data

In the Safari drop-down menu, select Preferences and click the Security icon.From there, select the desired security level in the "Accept cookies" area.

Disabling Cookies in your browser does not deprive you of access to the resources of the Website. Web browsers, by default, allow storing Cookies on the User's end device. Website Users can freely adjust cookie settings. The web browser allows you to delete cookies. It is also possible to automatically block cookies. Detailed information on this subject is provided in the help or documentation of the specific web browser used by the User. The User can decide not to receive Cookies by changing browser settings. However, disabling Cookies necessary for authentication, security or remembering User preferences may impact user experience, or even make the Website unusable.

5. Additional information

External links may be placed on the Website enabling Users to directly reach other website. Also, while using the Website, cookies may also be placed on the User’s device from other entities, in particular from third parties such as Google, in order to enable the use the functionalities of the Website integrated with these third parties. Each of such providers sets out the rules for the use of cookies in their privacy policy, so for security reasons we recommend that you read the privacy policy document before using these pages. We reserve the right to change this privacy policy at any time by publishing an updated version on our Website. After making the change, the privacy policy will be published on the page with a new date. For more information on the conditions of providing services, in particular the rules of using the Website, contracting, as well as the conditions of accessing content and using the Website, please refer to the the Website’s Terms and Conditions.

Nexocode Team


Want to unlock the full potential of Artificial Intelligence technology?

Download our ebook and learn how to drive AI adoption in your business.