World Leader in GPS
About The Client
The client is a world leader in advanced positioning solutions known for GPS, laser, optical, and inertial technologies and is renowned for integrating positioning technologies with application software, wireless communications, and services to provide complete commercial solutions. The client’s integrated solutions allow customers to manage and analyze complex information faster and easier, making them more productive, efficient, and profitable.
The client has many large multi-national customers who use their positioning solutions in the field. Each customer has many hundreds of sensors, IoT devices, and mobile endpoints which generate hundreds of thousands of messages per second on a continuous basis. Each customer also has their own unique requirements of utilizing these messages for business process management. Each customer has a variety of applications and databases that utilize information from these hundreds of endpoints. The client wanted to architect and build a common infrastructure that would be quick to deploy for various customers across various domains and still be scalable to meet the vast variety of requirements. The client was looking for guaranteed message delivery, scalability, durability, and repeatability.
AWS gives an option of flexible services which are designed to enable companies to build and deliver products more rapidly and reliably. This can be done using AWS and DevOps practices.
These services simplify the following:
- Simplify provisioning and managing infrastructure
- Deploying application code
- Automating software release processes
- Monitoring application and infrastructure performance
DevOps is the blend of social methods of insight, practices, and tools that builds an organization’s ability to deliver applications and services at high velocity. This helps evolving and improving products at a faster pace than organizations using traditional software development and infrastructure management processes. This speed enables organizations to better serve their customers and compete more effectively in the market.
Why 8K Miles
8K Miles with it’s unique combination of Cloud and Analytics expertise was the ideal partner.
- Incumbent Knowledge : 8K Miles has been servicing various divisions of the client for their various cloud needs successfully and hence had an intimate knowledge of the client’s business domain and organizational structure. This enabled 8K Miles to work efficiently and effectively as a partner to the client
- CloudEz : 8K Miles takes a platform approach to every project on the cloud. This CloudEz platform approach always ensures that our solutions bring security, repeatability and the necessary compliance
- Analytics : World class analytics leadership in handling big data, machine learning, prediction, and visualization. In addition to life sciences, we have helped many industries from cyber security to advertising adopt cutting edge analytics technologies
8K Miles recommended big data architecture for stream processing using Kafka on AWS. 8K Miles architected, implemented and is providing managed services for following system:
– Kafka platform was configured for processing the messages
- Secure Amazon Virtual Private Cloud (VPC) was instantiated with the necessary capacity to handle the current data sources and scalability for future data sizes running into multiple TB. Entire platform is set on Amazon EC2 and Amazon EBS.
- Kafka Cluster was setup with Multi Availability zone inside Amazon region for High Availability
- Kafka is configured with Zookeeper for
– Electing a controller – The controller is one of the brokers and is responsible for maintaining the leader/follower relationship for all the partitions. When a node shuts down, it is the controller that tells other replicas to become partition leaders to replace the partition leaders on the node that is going away. Zookeeper is used to elect a controller, make sure there is only one and elect a new one it if it crashes.- Cluster membership – to identify brokers that are alive and part of the cluster. This is also managed through ZooKeeper.
– Topic configuration – identify which topics exist, how many partitions each has, where are the replicas, who is the preferred leader, what configuration overrides are set for each topic.
HortonWorks Ambari is used for Hadoop management, It helps
– Make Hadoop management simpler by providing a consistent, secure platform for operational control. Ambari provides an intuitive Web UI as well as a robust REST API, which is particularly useful for automating cluster operations. With Ambari, Hadoop operators get the following core benefits:- Simplified Installation, Configuration and Management. Easily and efficiently create, manage and monitor clusters at scale. Takes the guesswork out of configuration with Smart Configs and Cluster Recommendations.
– Centralized Security Setup. Reduce the complexity to administer and configure cluster security across the entire platform.
– Full Visibility into Cluster Health. Ensure your cluster is healthy and available with a holistic approach to monitoring.
With 8K Miles architected and developed solution, the client was able to deploy this messaging as a service solution to their customer within a couple of months. 8K Miles also manages the big data cluster providing operational efficiency and value to the customer.
About 8K Miles