Stories by Ankit Bourasi on Medium

Hello guys! 👋 If you’re reading this, you’re probably curious about Apache Kafka. Don’t worry if you’ve never heard of it before — we’re going to start from scratch and make this journey fun and easy to understand.
I am Ankit and you’re learning Kafka, Let’s dive into this!
<figure><img alt="" src="https://cdn-images-1.medium.com/max/702/0*rct3WLEDJT6voe40.png"></figure>Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
Apache Kafka is a tool that helps move and process large amounts of data quickly and reliably between different parts of a computer system
Let’s understand by simple example of Twitter:
<ol><li>A User does the following actions:</li></ol>
<ul>
<li>user posts a tweet</li>
<li>another user likes the tweet</li>
<li>a third user retweets it</li>
</ul>
All of this Twitter actions acts as events in the applications, and these actions sent to Kafka because twitter has 368 million active Users,
so it very difficult to process and manage those events that’s why companies use Kafka for high volume of Data.
<ul>
<li>Each of these actions generates an “event” that gets sent to Kafka</li>
<li>Kafka organizes these events into “topics” (e.g., “new_tweets”, “likes”, “retweets”)</li>
</ul>
The Twitter Components
<ul>
<li>Tweet Storage Service: Saves new tweets to a database</li>
<li>Notification Service: Sends notifications to users</li>
<li>Analytics Service: Tracks trending topics and user engagement</li>
<li>Timeline Service: Updates users’ timelines</li>
</ul>
2. How it works:
<ul>
<li>These services “subscribe” to relevant Kafka topics</li>
<li>When an event occurs, Kafka quickly distributes it to all subscribed services</li>
<li>Each service processes the event as needed</li>
</ul>
For example, when a tweet is posted:
<ul>
<li>Tweet Storage Service saves the tweet</li>
<li>Notification Service notifies followers</li>
<li>Analytics Service updates trending topics</li>
<li>Timeline Service adds the tweet to followers’ timelines</li>
</ul>
Kafka allows all of this to happen simultaneously and efficiently, even with millions of users and tweets. It ensures that no events are lost, even if a service temporarily goes down, and can handle massive amounts of data in real-time.
<h3>Understand Kafka Terminologies</h3>
<ol>
<li>
Kafka Cluster: A Kafka Cluster is a group of Kafka servers (or machines) that work together to manage and store streams of data</li>
<li>
Kafka Broker (or Kafka Server): A Kafka Broker is one individual server in the Kafka Cluster. It stores data and handles requests from producers and consumers.</li>
<li>
Kafka Topic: A Kafka Topic is a category or you can think of a Table of SQL databases in which messages are sent.</li>
</ol>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/550/0*WsXboJ7j1aR-sO_G"></figure>4. Partition: A Partition is a smaller piece of a topic. Each topic can be divided into several partitions to allow for parallel processing and better performance.
You can think of partitions as sections within a topic where messages are stored in a particular order.
<figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*V_FIUKgr-VrFkqPQ"></figure>5. Segment: A Segment is a file within a partition that holds a portion of the messages. As new messages come in, they are added to segments, which are like chapters in a book.
When a segment gets too large, it’s split into new segments.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/526/0*j6by--e33sOs7l5H"></figure>6. Producer: A Producer is an application or component that sends messages to a Kafka topic. Think of it as a sender or publisher that creates and pushes data into Kafka.
7. Consumer: A Consumer is an application or component that reads messages from a Kafka topic. It acts like a reader or subscriber that retrieves and processes data from Kafka.
8. Messages: Messages are the actual pieces of data sent between producers and consumers. They contain the information being transmitted, like a text message or a data record. Messages are organized and stored in topics and partitions.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BJzVwGksYHedBa62"></figure><h4>Putting It All Together:</h4>
<ul>
<li>
Kafka Cluster: A collection of servers working together.</li>
<li>
Kafka Broker: An individual server in the cluster.</li>
<li>
Kafka Topic: A category for organizing messages.</li>
<li>
Partition: A sub-section within a topic for organizing messages.</li>
<li>
Segment: A file that stores a portion of messages in a partition.</li>
<li>
Producer: The entity that sends messages to a topic.</li>
<li>
Consumer: The entity that reads messages from a topic.</li>
<li>
Messages: The actual data being sent and received.</li>
</ul>
<h4>Kafka Architecture -</h4>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hiz6liIvh9iYIWi5eqPaiw.png"></figure>Kafka is a system that efficiently manages and processes streams of data across a network of servers, allowing different applications to publish and consume data in real-time.
<h4>Download Kafka</h4>
You need to download Kafka binaries from the official website. Download Kafka:
<ul>
<li>Go to the <a href="https://kafka.apache.org/downloads">Apache Kafka Downloads page</a>.</li>
<li>Download the latest binary version.</li>
<li>Extract the downloaded ZIP or TAR file to your desired location. rename folder to kafka
</li>
</ul>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/475/0*7fri8LEgyxC3nDj2"></figure><blockquote>
Note: To install kafka, you need Java 8 installed in your system.
</blockquote>
<h4>Start the Kafka server</h4>
<ol><li>Go to your kafka directory, Open terminal in current directory such as Command Prompt[CMD], Powershell, Bash, Tmux, Warp.
</li></ol>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/726/0*YTvuWczWj5ECUt0O"><figcaption>Bash Terminal Linux</figcaption></figure>2. Before start kafka server, we need to start the ZOO KEEPER Kafka uses ZooKeeper to manage the Kafka cluster. ZooKeeper is included with the Kafka download, but it needs to be started before Kafka can run.
<blockquote>You don’t need to install ZooKeeper separately as it’s bundled with Kafka.</blockquote>
<h4>Start ZooKeeper</h4>
Kafka requires ZooKeeper to run, so you need to start it first. Run the following command from the Kafka installation directory:
<h4>Kafka with Zookeeper For [Linux] User</h4>
Run the following commands in order to start all services in the correct order:
<pre># Start the ZooKeeper service $ bin/zookeeper-server-start.sh config/zookeeper.properties</pre>
Open another terminal session and run:
<pre># Start the Kafka broker service $ bin/kafka-server-start.sh config/server.properties</pre>
Once all services have successfully launched, you will have a basic Kafka environment running and ready to use.
<h4>Kafka with Zookeeper For [Windows] User</h4>
Run the following commands in CMD in order to start all services in the correct order:
<pre># Start the ZooKeeper service $ bin/windows/zookeeper-server-start.sh config/zookeeper.properties</pre>
Open another terminal session in CMD and run:
<pre># Start the Kafka broker service $ bin/windows/kafka-server-start.sh config/server.properties</pre>
Once all services have successfully launched, you will have a basic Kafka environment running and ready to use.
<blockquote>
zookeeper-server-start file start the zookeeper, zookeeper.properties is configuration file for zookeeper</blockquote>
<blockquote>
kafka-server-start file start the kafka server, server.properties is a configuration file for kafka</blockquote>
You should see these type of log in Zookeeper terminal, it indicates that Zookeeper has started successfully.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kL8VSzwbzZYuJXMB"></figure>Kafka Server Logs if successfully started:-
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_iX55FHz6cx7gJY-"></figure><h3>Congratulation You have successfully start the Kafka environment running and ready to use.</h3>
<h4>For more understanding and hands on experience visit this blog</h4>
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=426c270d6c18" width="1" height="1" alt="">

Kafka for Beginners

Today I won't make you a fool by telling you some advantages of Python and JavaScript, Instead, I’ll tell you why these languages are used more often rather than other popular languages.
Let’s break it down in simple terms and see what makes them stand out.
<h3>Why Python and JavaScript Are So Popular</h3>
Python and JavaScript are everywhere these days. You’ll find them in web development, data science, machine learning, and more. But why are these two languages so widely used? Let’s break it down in simple terms and see what makes them stand out.
<h3>How Do They Work?</h3>
Let’s look at some popular tools and libraries built with Python and JavaScript to understand better:
<h4>1. Node.js</h4>
<ul>
<li>
What It Is: Node.js is a way to run JavaScript on the server side.</li>
<li>
How It Works: It uses Google’s V8 JavaScript engine for fast performance. Behind the scenes, Node.js is powered by C++ which helps it talk to the operating system and handle tasks efficiently.</li>
</ul>
<h4>2. TensorFlow</h4>
<ul>
<li>
What It Is: TensorFlow is a library for building machine learning models.</li>
<li>
How It Works: TensorFlow uses Python for most tasks because Python is easy to write and understand. But the heavy lifting is done in C++, which makes the calculations fast.</li>
</ul>
<h4>3. PyTorch</h4>
<ul>
<li>
What It Is: PyTorch is another library for machine learning, known for its ease of use.</li>
<li>
How It Works: Like TensorFlow, PyTorch uses Python for the user-friendly parts. Its core operations are handled by C++ for better performance.</li>
</ul>
<h4>4. Pandas</h4>
<ul>
<li>
What It Is: Pandas is a library for working with data in Python .</li>
<li>
How It Works: Pandas relies on NumPy for fast numerical calculations. NumPy itself is built on C, which makes data operations quick and efficient.</li>
</ul>
<h4>5. NumPy</h4>
<ul>
<li>
What It Is: NumPy is a fundamental library for numerical computing in Python.</li>
<li>
How It Works: NumPy’s core is written in C and Fortran, which means it can perform complex calculations rapidly. Python acts as the friendly interface for these powerful computations.</li>
</ul>
<h3>Why Python and JavaScript?</h3>
<h4>1. Ease of Use</h4>
<ul><li>Both languages are designed to be simple and intuitive. This makes it easier for developers to write and understand code. Whether you’re doing data analysis or building a website, Python and JavaScript make it straightforward.</li></ul>
<h4>2. Integration with Fast Languages</h4>
<ul><li>Python and JavaScript act as wrappers around lower-level languages like C and C++. This means developers can use Python or JavaScript to build applications quickly, while still benefiting from the speed and efficiency of C and C++.</li></ul>
<h4>3. Rapid Development</h4>
<ul><li>With extensive libraries and frameworks available, both Python and JavaScript allow for quick development and testing. This makes it possible to build and improve features at a fast pace.</li></ul>
<h3>Real-World Impact</h3>
The popularity of Python and JavaScript isn’t just because of their ease of use. It’s also due to their ability to handle complex tasks efficiently by leveraging powerful underlying technologies. This combination of user-friendly interfaces and high performance is why these languages are so popular and widely adopted.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=fe477753a531" width="1" height="1" alt="">

Python and JavaScript. Why are used as wrappers?

Hello, guys as you know data science is one of the most in-demand jobs in the next 5–10 years. there are tons of advantages if you can learn the skills of data science and machine learning and go into the job market to find great opportunities for yourself and that is why here 5 FREE courses that you can take from google and get certificates for them.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*RMN0WZVVuHmGU5yjcKRRIg.jpeg"></figure>1. Data science foundation — Google
If you are new to data science then you can start from this course, here you will learn about data science foundations like what is data science? data science life cycle? how to become a data scientist? and more,
Duration: 1.5 hrs,
<a href="https://learndigital.withgoogle.com/digitalunlocked/course/data-science-foundations">Data Science Foundations</a>
2. Learn python basics for data analysis — Google
This course will help you understand the basics of python if you are completely new to programming but you want to start data science you first start with a language this course is very helpful for you.
Duration: 6 hrs,
<a href="https://learndigital.withgoogle.com/digitalunlocked/course/learn-python-basics-for-data-analysis">Grow with Google: Get the right skills to reach new opportunities</a>
3. Data science with python — Google
This is a much more in-depth course, which will teach you about the advanced stuff like data wrangling, data exploration, visualization, web scraping, and more, that you can do with python in the field of data science
Duration: 8 hrs,
<a href="https://learndigital.withgoogle.com/digitalgarage/course/data-science-with-python">Data Science with Python</a>
4. Understand the basics of machine learning — Google
here you will learn basic stuff about machine learning and how it works and how to understand the basic concept behind machine learning
Duration: 19 min,
<a href="https://learndigital.withgoogle.com/digitalgarage/course/machine-learning-basics">Introduction to Machine Learning</a>
5. Machine learning crash course — Google
Google’s fast-paced, practical introduction to machine learning, features a series of lessons with video lectures, real-world case studies, and hands-on practice exercises.
Duration: 15 hrs,
<a href="https://developers.google.com/machine-learning/crash-course">Machine Learning | Google for Developers</a>
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=383ee6c18436" width="1" height="1" alt="">

GOOGLE Launched 5 FREE Data Science Courses! (With Certificates)

<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mBQ_qAWBW2hrNOLES1Y5Zw.jpeg"></figure>Photo by <a href="https://unsplash.com/@yuyeunglau">Yuyeung Lau</a> on<a href="https://unsplash.com/photos/dp9Jrww_BRs"> Unsplash</a>
You probably know artificial intelligence (AI) as the technology that enables self-driving cars or helpful virtual assistants like Siri and Alexa. But AI is so much more than that. In fact, AI is poised to revolutionize almost every industry in the coming years. In this blog post, you will learn everything you need to know about AI, from what it is and how it’s currently being used to its potential future applications and risks.
<h3>What is Artificial Intelligence?</h3>
AI is an area of computer science dedicated to the concept of making machines that can think and act like humans. Over the years, the term has come to refer to many different subfields of computer science, each trying to achieve the same goal in different ways. Whatever the subfield, the ultimate goal is always the same: to create a machine that can think and act like a human. Computer scientists use the terms artificial, synthetic, or machine intelligence to refer to computer programs that can solve problems on their own by mimicking human cognition. While artificial intelligence is not a single thing, it generally refers to computer systems that can do what we call “human intelligence” tasks like analyzing data, understanding language, creating and designing content, making autonomous decisions, and solving complex problems.
<h3>How is AI currently being used?</h3>
- Natural Language Processing (NLP) — NLP uses AI to understand the meaning behind words. This is what allows your virtual assistants to understand what you say and respond appropriately. — Machine Learning — ML is the subfield of AI that enables computers to improve as they get more data. For example, if a computer is trying to recognize images, it will look at thousands of images and get feedback from human trainers about what’s in the images. — Computer Vision — CV is the subfield of AI that enables computers to understand the images they see through image recognition. This technology can be used for things like autonomous drones and robots, visual search engines, and medical imaging. — Computer-driven Strategies — This refers to AI-powered trading algorithms that enable investors to execute trades in seconds. While humans can operate with a level of risk, computers do not have this capacity. This will enable computers to make more accurate and faster decisions.
<h3>Why is everyone talking about AI now?</h3>
There are a few reasons why AI has become such a hot topic in the last few years. First, the technology is truly booming: The number of AI startups is increasing every year, and companies are pouring billions of dollars into AI research and development. — Another factor is that AI has the potential to revolutionize many industries, so the topic attracts a lot of attention and inspires both excitement and anxiety about the future. — A final reason is that the AI community has been criticized for not being transparent about the technology and its potential impact. — This has led many stakeholders to demand more and better communication about AI and its implications.
<h3>How will AI change the world?</h3>
AI will likely be used to solve some of the world’s biggest and most pressing issues, such as the environment, health and wellness, and the economy. — Environmental issues — AI will be used to monitor and regulate pollution, deforestation, and climate change. It will be used to create self-sustaining green cities, and it will even be used to create new materials like hybrid plants and food. — Health and wellness — AI will be used to create new drugs, better diagnostic tools and treatments, and even new foods that improve wellness. It will be used to create personalized wellness programs, and it will even be used to create personalized climates. — Economy — AI will be used to create new business opportunities and investment strategies, and it will be used to create better programs to help those in need. It will be used to create new payment systems, and it will even be used to create better communication platforms.
<h3>Is there anything to fear from AI?</h3>
Yes, there is one thing to fear from AI, and it’s one that’s been talked about for decades: If a machine is able to think and make decisions like a human, does that mean we no longer have a place? This has led some to argue that AI will lead to the end of the human race. — This kind of thinking is extremely dangerous and will cause a lot of fear, which will only hold back important scientific developments. — Instead of fearing AI, we should focus on making sure that it is developed ethically and that its benefits are distributed evenly across society. — This means not only developing AI but also making sure that its benefits extend to everyone.
<h3>Bottom line</h3>
Artificial intelligence is an exciting and important area of computer science. While the term can refer to many different subfields, all of these subfields attempt to give machines the capacity to think and act like humans. And while there are legitimate reasons to be concerned about AI, the benefits of this technology are likely to outweigh the risks. That said, AI is still in its early stages, and there’s a long way to go before computers become truly intelligent. In the meantime, we can use AI to make our lives easier and help us tackle some of society’s biggest problems.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=3e858bc3dd20" width="1" height="1" alt="">

Informative

Recent Articles