Top Apache Flink Interview Questions and Answers (2024)

In this post, questions from Apache Flink Interviews will be answered for Experienced and Freshers. We're trying to share our experience and learn how to help you make progress in your career.

1. What is Apache Flink?
2. Explain Apache Flink Architecture?
3. Explain the Apache Flink Job Execution Architecture?
4. What is Unbounded streams in Apache Flink?
5. What is Bounded streams in Apache Flink?
6. What is Dataset API in Apache Flink?
7. What is DataStream API in Apache Flink?
8. What is Apache Flink Table API?
9. What is Apache Flink FlinkML?
10. What are the differences between Apache Hadoop, Apache Spark and Apache Flink?

Q: What is Apache Flink?
Ans:

The Apache Software Foundation created Apache Flink, an open-source, unified stream-processing and batch-processing framework. Apache Flink is built around a distributed streaming data-flow engine written in Java and Scala. Flink runs every dataflow programm in a data-parallel and pipelined fashion.

Q: Explain Apache Flink Architecture?
Ans:

Apache Flink is based on the Kappa architecture. The Kappa architecture uses a single processor - stream, who accepts all information as a stream, and the streaming engine processes data in real-time. Batch data in kappa architecture is a form of streaming data.

The majority of big data frameworks are built on the Lambda architecture, which uses different processors for batch and streaming data. In Lambda architecture, batch and stream views have different codebases. The codebases must be combined in order to query and retrieve the results. Maintaining different codebases/views and combining them is a hassle, but the Kappa architecture fixes this problem by having only one real-time view, which eliminates the need for codebase merging.

The core principle of the Kappa architecture is to manage batch and real-time data via a single stream processing engine.

That is not to say that Kappa architecture is preferred to Lambda architecture; it is entirely dependent on the use-case and application to determine which architecture is optimal.

Q: Explain the Apache Flink Job Execution Architecture?
Ans:

The Apache Flink job execution architecture is shown in the diagram below.

1. Program

   It is a piece of code that is executed on the Flink Cluster.

2. Client

   It is in charge of taking code from the given programm and creating a job dataflow graph, which is then passed to JobManager. It also retrieves the Job data.

3. JobManager

   It is responsible for generating the execution graph after obtaining the Job Dataflow Graph from the Client. It assigns the job to TaskManagers in the cluster and monitors its execution.

4. TaskManager

   It is in charge of executing all of the tasks assigned to it by JobManager. Both TaskManagers execute the tasks in their respective slots in the specified parallelism. It is in charge of informing JobManager about the status of the tasks.

Q: What is Unbounded streams in Apache Flink?
Ans:

Any type of data is produced as a stream of events. Data can be processed as unbounded or bounded streams.

Unbounded streams have a beginning but no end. They do not end and continue to provide data as it is produced. Unbounded streams should be processed continuously, i.e., events should be handled as soon as they are consumed. Since the input is unbounded and will not be complete at any point in time, it is not possible to wait for all of the data to arrive.

Processing unbounded data sometimes requires that events are consuming in a specific order, such as the order in which events arrives, to be able to reason about result completeness.

Q: What is Bounded streams in Apache Flink?
Ans:

Bounded streams have a beginning and an end point. Bounded streams could be processed by consuming all data before doing any computations. Ordered ingestion is not needed to process bounded streams since a bounded data set could always be sorted. Processing of bounded streams is also called as batch processing.

Q: What is Dataset API in Apache Flink?
Ans:

The Apache Flink Dataset API is used to do batch operations on data over time. This API is available in Java, Scala, and Python. It may perform various transformations on datasets such as filtering, mapping, aggregating, joining, and grouping.

 DataSet<Tuple2<String, Integer>> wordCounts = text
       .flatMap(new LineSplitter())
       .groupBy(0)
       .sum(1);

Q: What is DataStream API in Apache Flink?
Ans:

The Apache Flink DataStream API is used to handle data in a continuous stream. On the stream data, you can perform operations such as filtering, routing, windowing, and aggregation. On this data stream, there are different sources such as message queues, files, and socket streams, and the resulting data can be written to different sinks such as command line terminals. This API is supported by the Java and Scala programming languages.

 DataStream<Tuple2<String, Integer>> dataStream = env
     .socketTextStream("localhost", 9091)
     .flatMap(new Splitter())
     .keyBy(0)
     .timeWindow(Time.seconds(7))
     .sum(1);

Q: What is Apache Flink Table API?
Ans:

Table API is a relational API with an expression language similar to SQL. This API is capable of batch and stream processing. It is compatible with the Java and Scala Dataset and Datastream APIs. Tables can be generated from internal Datasets and Datastreams as well as from external data sources. You can use this relational API to perform operations such as join, select, aggregate, and filter. The semantics of the query are the same if the input is batch or stream.

 val tableEnvironment = TableEnvironment.getTableEnvironment(env)

 // register a Table
 tableEnvironment.registerTable("TestTable1", ...);

// create a new Table from a Table API query
 val newTable2 = tableEnvironment.scan(TestTable1).select(...);

Q: What is Apache Flink FlinkML?
Ans:

FlinkML is the Flink Machine Learning (ML) library. It is a new initiative in the Flink community, with an expanding list of algorithms and contributors. FlinkML aims to include scalable ML algorithms, an easy-to-use API, and tools to help reduce glue code in end-to-end ML systems. Note: Flink Community has planned to delete/deprecate the legacy flink-libraries/flink-ml package in Flink1.9, and replace it with the new flink-ml interface proposed in FLIP39 and FLINK-12470.

Q: What are the differences between Apache Hadoop, Apache Spark and Apache Flink?
Ans:

	Hadoop	Spark	Flink
Data Processing	Apache Hadoop built for batch processing.	Apache Spark is also a part of Hadoop Ecosystem. It is mostly a batch processing system, but it also supports stream processing.	Apache Flink offers a single runtime for both streaming and batch processing.
Streaming Engine	Apache Hadoop has NO streaming engine. In Hadoop, map-reduce is a batch-processing tool. It takes a huge data set as input, processes it, and outputs the result all at once.	Data streams are processed in micro-batches through Apache Spark Streaming.	Apache Flink is the real streaming engine.Streams are used for workloads such as streaming, SQL, micro-batch, and batch. A batch is a limited collection of streamed results.
Performance	Apache Hadoop performance is slower when compared Spark and Flink, because it only supports batch processing and not streaming.	Apache Spark Stream processing is less effective than Apache Flink as it uses micro-batch processing.	Apache Flink performance is excellent as compared to Hadoop and Spark.
Memory management	Disk Based- Configurable Memory management.	JVM Managed- configurable memory management, since Spark 1.6 support automating memory management	Active Managed - automatic memory management, own memory management system, separate from Java's garbage collector
Fault tolerance	MapReduce is extremely fault tolerant. In the case of a Hadoop failure, there is no need to restart the application from scratch.	Apache Spark Streaming restores lost work without the need for additional code or configuration.	Apache Flink is based on Chandy-Lamport distributed snapshots. It is lightweight, allowing for high throughput rates while still providing good accuracy guarantees.
SQL Support	Hive, Impala	SparkSQL	Table API and SQL
Machine Learning Support	NA	Apache Spark support SparkML.	Apache Flink support FlinkML.
Programming Languages	It supports Java, C, C++, Ruby, Groovy, Perl, Python	Apache Spark supports Java, Scala, python and R.	Apache Flink supports Java and Scala.