DAY 4 - Industry Connect

What is CI, CD, CT? 

Continuous integration (CI), continuous delivery/deployment (CD), and continuous testing (CT) accelerate the code release process and help with deployment.

IT organizations are increasingly turning to DevOps practices to improve value delivery for the customer. When DevOps implements CI, CD, and CT methods correctly, releases become more reliable and error-free. This, in turn, leads to higher productivity, efficiency, and customer satisfaction.

Background

We will talk about each of these – CI, CD, and CT – in detail and their role in DevOps. They are the main methods to measure successful DevOps practice. We will talk about how they make DevOps practices more trouble-free.

The discussion around them is important because now, with DevOps, developers receive feedback from stakeholders and tools that systematically monitor and test their code in the pipeline. Enter: CI, CD, and CT.

Summary

What to expect:

• What is CI in DevOps?
• What is CD in DevOps?
• What is CT in DevOps?
• Using CI, CD, and CT in a DevOps Pipeline

What Is CI in DevOps?

CI allows developers to write, update, or fix a feature, then commit a code to the central code repository reliably, multiple times a day. Each update triggers an automated build and testing sequence for a given project, ensuring that code changes that merge into the repository are reliable.

It’s a critically important step to increasing deployment efficiency. CI continuously integrates code changes into the existing codebase to quickly identify and remediate any conflicts between different developers’ code changes.

If a development team is not making frequent commits to a common branch in a shared source code repository, they are not successfully doing continuous integration. Additionally, if the test and build processes are automated but developers work on isolated branches that are not often integrated into a shared branch, that is also not continuous integration.

What Is CD in DevOps?

CD is the implementation of automating the entire software release process. Once code has passed all of the tests, deployment becomes the last step in the process. In a DevOps pipeline, CD often is referred to as continuous delivery but can also mean continuous deployment.

Below is an explanation of both.

Continuous Delivery vs. Continuous Deployment

Continuous delivery follows these steps:

• Puts a change in the staging environment and release schedule,
• A person manually approves this code,
• Finally, it is deployed.

Continuous deployment automatically deploys code to production once it passes tests.

These two are not mutually exclusive, but they include each other. Their overall goal is the same – make software development and release processes faster and more robust. The key difference between them is the scope of automation applied.

With continuous delivery, the process allows developers to be more productive by deploying a new release at any time with a few clicks. On the other hand, continuous deployment is a step up from continuous delivery as every change in the source code is deployed to production automatically. This deployment does not require approval from a developer.

What Is CT in DevOps?

While CI and CD are crucial in accelerated product releases, CT brings the quality factor into these frequent releases. In the CI/ CD pipeline or DevOps pipeline, sometimes CT is omitted even though it is a critical component in any pipeline. The real benefits of adopting CI/ CD are not truly felt if there is a lack of automated testing.

CT is a primary enabler of continuous feedback. Not only is it considered the first step in the right direction when embarking on a DevOps journey, but it drives software delivery through the software development cycle (SDLC) by being a continuous feedback mechanism. It allows DevOps teams to enable deployment and post-deployment testing in local environments.

Using CI, CD, CT in a DevOps Pipeline

A CI, CD, CT pipeline automates the entire deployment. It protects against substandard or error-prone changes to the codebase and early detection of code defects.

Benefits of Integrating CI, CD, CT

Manual software deployment is susceptible to human error. That is why it’s important to have these three processes run smoothly. One process cannot be successful without the other two working seamlessly throughout the delivery cycle. Although each has different roles and responsibilities, they depend on one another for a quality deployment.

• By automating the steps leading up to a product release, an organization can ensure that both the code and the deployment process work efficiently and correctly.
• When developers or other teams find a defect, they can rely on the CI, CD, CT pipeline to automate, for example, a rollback to the previous code version.
• When all three are combined, they can significantly help increase the most important goals of any development team: velocity and quality.
• By connecting CT closely with CI/CD, developers can improve operational agility and accelerate time-to-market.

Advice on How to Integrate CI, CD, CT Correctly

1. The best thing that developers can do is making sure that their automated test suite is comprehensive and stable. This will make them feel safe to implement every passed CI build to staging and then production status without a long manual quality assurance check.
2. Pay close attention to CI speed and optimize when available. Otherwise, developer productivity can shrink due to a lack of focus and frequent context switching.
3. Communication is something vital that CI enables; teams are able to be agile and on the same page with a project.
4. Trust is something that CD upholds. CD eliminates the element of doubt by automating and streamlining all processes leading up to deployment.
5. Honesty is the best policy for developers when it comes to CT. If CT is utilized when developing apps, it will prevent more significant issues from happening once the app is live, keeping developers honest about the code status.

What are the benefits for a team in following DevOps practices? 

DevOps is an evolving philosophy and framework that encourages faster, better application development and faster release of new or revised software features or products to customers.

The practice of DevOps encourages smoother, continuous communication, collaboration, integration, visibility, and transparency between application development teams (Dev) and their IT operations team (Ops) counterparts.

This closer relationship between “Dev” and “Ops” permeates every phase of the DevOps lifecycle: from initial software planning to code, build, test, and release phases and on to deployment, operations, and ongoing monitoring. This relationship propels a continuous customer feedback loop of further improvement, development, testing, and deployment. One result of these efforts can be the more rapid, continual release of necessary feature changes or additions.

Some people group DevOps goals into four categories: culture, automation, measurement, and sharing (CAMS), and DevOps tools can aid in these areas. These tools can make development and operations workflows more streamlined and collaborative, automating previously time-consuming, manual, or static tasks involved in integration, development, testing, deployment, or monitoring.

Why DevOps matters

Along with its efforts to break down barriers to communication and collaboration between development and IT operations teams, a core value of DevOps is customer satisfaction and the faster delivery of value. DevOps is also designed to propel business innovation and the drive for continuous process improvement.

The practice of DevOps encourages faster, better, more secure delivery of business value to an organization’s end customers. This value might take the form of more frequent product releases, features, or updates. It can involve how quickly a product release or new feature gets into customers’ hands—all with the proper levels of quality and security. Or, it might focus on how quickly an issue or bug is identified, and then resolved and re-released.

Underlying infrastructure also supports DevOps with seamless performance, availability, and reliability of software as it is first developed and tested then released into production.

DevOps methods

There are a few common DevOps methods that organizations can use to speed and improve development and product releases. They take the form of software development methodologies and practices. Among the most popular ones are Scrum, Kanban, and Agile:

• Scrum. Scrum defines how members of a team should work together to accelerate development and QA projects. Scrum practices include key workflows and specific terminology (sprints, time boxes, daily scrum [meeting]), and designated roles (Scrum Master, product owner).
• Kanban. Kanban originated from efficiencies gained on the Toyota factory floor. Kanban prescribes that the state of software project work in progress (WIP) be tracked on a Kanban board.
• Agile. Earlier agile software development methods continue to heavily influence DevOps practices and tools. Many DevOps methods, including Scrum and Kanban, incorporate elements of agile programming. Some agile practices are associated with greater responsiveness to changing needs and requirements, documenting requirements as user stories, performing daily standups, and incorporating continuous customer feedback. Agile also prescribes shorter software development lifecycles instead of lengthy, traditional “waterfall” development methods.

DevOps toolchain

Followers of DevOps practices often use certain DevOps-friendly tools as part of their DevOps “toolchain.” The goal of these tools is to further streamline, shorten, and automate the various stages of the software delivery workflow (or “pipeline”). Many such tools also promote core DevOps tenets of automation, collaboration, and integration between development and operations teams. The following shows a sample of tools used at various DevOps lifecycle stages.

• Plan. This phase helps define business value and requirements. Sample tools include Jira or Git to help track known issues and perform project management.
• Code. This phase involves software design and the creation of software code. Sample tools include GitHub, GitLab, Bitbucket, or Stash.
• Build. In this phase, you manage software builds and versions, and use automated tools to help compile and package code for future release to production. You use source code repositories or package repositories that also “package” infrastructure needed for product release. Sample tools include Docker, Ansible, Puppet, Chef, Gradle, Maven, or JFrog Artifactory.
• Test. This phase involves continuous testing (manual or automated) to ensure optimal code quality. Sample tools include JUnit, Codeception, Selenium, Vagrant, TestNG, or BlazeMeter.
• Deploy. This phase can include tools that help manage, coordinate, schedule, and automate product releases into production. Sample tools include Puppet, Chef, Ansible, Jenkins, Kubernetes, OpenShift, OpenStack, Docker, or Jira.
• Operate. This phase manages software during production. Sample tools include Ansible, Puppet, PowerShell, Chef, Salt, or Otter.
• Monitor. This phase involves identifying and collecting information about issues from a specific software release in production. Sample tools include New Relic, Datadog, Grafana, Wireshark, Splunk, Nagios, or Slack.

DevOps practices

DevOps practices reflect the idea of continuous improvement and automation. Many practices focus on one or more development cycle phases. These practices include:

• Continuous development. This practice spans the planning and coding phases of the DevOps lifecycle. Version-control mechanisms might be involved.
• Continuous testing. This practice incorporates automated, prescheduled, continued code tests as application code is being written or updated. Such tests can speed the delivery of code to production.
• Continuous integration (CI). This practice brings configuration management (CM) tools together with other test and development tools to track how much of the code being developed is ready for production. It involves rapid feedback between testing and development to quickly identify and resolve code issues.
• Continuous delivery. This practice automates the delivery of code changes, after testing, to a preproduction or staging environment. An staff member might then decide to promote such code changes into production.
• Continuous deployment (CD). Similar to continuous delivery, this practice automates the release of new or changed code into production. A company doing continuous deployment might release code or feature changes several times per day. The use of container technologies, such as Docker and Kubernetes, can enable continuous deployment by helping to maintain consistency of the code across different deployment platforms and environments.
• Continuous monitoring. This practice involves ongoing monitoring of both the code in operation and the underlying infrastructure that supports it. A feedback loop that reports on bugs or issues then makes its way back to development.
• Infrastructure as code. This practice can be used during various DevOps phases to automate the provisioning of infrastructure required for a software release. Developers add infrastructure “code” from within their existing development tools. For example, developers might create a storage volume on demand from Docker, Kubernetes, or OpenShift. This practice also allows operations teams to monitor environment configurations, track changes, and simplify the rollback of configurations.

Benefits of DevOps

DevOps proponents describe several business and technical benefits, many of which can result in happier customers. Some benefits of DevOps include:

• Faster, better product delivery
• Faster issue resolution and reduced complexity
• Greater scalability and availability
• More stable operating environments
• Better resource utilization
• Greater automation
• Greater visibility into system outcomes
• Greater innovation

History of DevOps

Many DevOps methods for streamlining software development and deployment have an early basis in agile software development and lean programming. But DevOps originally evolved from several grassroots movements to harmonize the activities of developers and their operations team counterparts.

The early 2000s saw the need to maintain availability of popular websites such as Google and Flickr against massive hits. This need led to the use of software reliability engineers (SREs)—operations people working closely with developers to ensure that the sites would keep running after code was released into production.

In 2009, Flickr engineers John Allspaw and Paul Hammond presented their own DevOps-like methodology at a conference. Their presentation was entitled “10+ Deploys per Day: Dev and Ops Cooperation at Flickr.” The same year, Patrick Debois organized the first “DevOps Day” in Belgium. A #DevOps hashtag was also incorporated and gained momentum as more DevOps Days were held around the world.

Over the coming years, industry and open-source tools and frameworks were developed and proposed to further the goals of DevOps.

NetApp and DevOps

NetApp helps promote better business outcomes with DevOps: Operations teams deliver automated infrastructure with less engineering, and developers create in reliable and predictable environments with less friction. With NetApp® technology, you can confidently deliver the services and capabilities your organization needs to maximize developer productivity in both on-premises and cloud environments.

NetApp makes it easy for developers, testing, QA, and operations teams to consume infrastructure resources (such as persistent storage volumes) as code, all from within familiar tools in the DevOps pipeline. Examples include NetApp storage APIs and IAC integrations with tools such as Puppet, Ansible, Docker, Kubernetes, and OpenShift.

But NetApp doesn’t just make it easy for developers to consume storage as code. NetApp technology also allows operations to confidently provision and deliver resources to development and testing personnel. Developers and testers can then accelerate coding and test cycles by self-provisioning fast, time-saving storage features, such as snapshots or clones. With these features, copies of real-time production data or code sets can be provisioned as code in mere seconds or minutes to speed development and QA workflow. Learn more about why NetApp is made for DevOps.

Why NetApp is made for DevOps

NetApp commitment to the community extends into thePub. This developer community encourages collaboration, shares tips and tricks, and debates ideas around the latest DevOps practices and protocols.

What is the purpose of GitHub? 

The main purpose of GitHub is version control of your code. There are many other benefits and features too but the single most important purpose is that - version control.

Imagine you are writing a program. You wrote something quick and dirty that works and solves the problem. You commit it into a GitHub repository then and there.

Then you move forward with giving a proper shape and elegance to your quick and dirty solution. You make a small change that still works but makes your solution a bit better in some way. You commit the change into the same repo in GitHub then and there.

Move on to the next change. You see that the changes you made now broke something that was working earlier and it is a soup now. You don't know what to do and how to untangle the code now. No problem. You can checkout from GitHub to the last working version and restart from there. Imagine you did not have GitHub. You would be forever worried about what was the last working version forever.

Now imagine you have 2 possible ways to implement the next change. You are unsure of which way to choose in the final production version. No problem. Make 2 branches in the repo, one for each method, and progress implementing both methods. Experiment and see which method is better and merge that branch into the master branch and discard the other branch.

Now imagine your friend also gets interested in your work and wants to contribute. You can ask your friend to fork your GitHub repo and start contributing in his own fork. When your friend is ready to push the code into your main branch he can send you a “pull request” so that you can pull in his solution into some branch of your repo. You can review his changes and ask him to rework or pull in his changes into your repo.

So like that, GitHub enables you precisely control how your code grows. Of course, it is the Git in the background that provides the engine and powers all this. GitHub allows you to easy create and manage these work flows on top of Git. But for simplicity you can consider GitHub as a version control tool.

You can of course do a lot more too - provide documentation, track issues, manage a small sized project, etc., and what i said above are the very basics.

A budding programmer like you should be using GitHub to store all the code you write for your pet projects. That way you can grow your portfolio in a stable structured way and share them easily too.

Advantages 

As some people think, GitHub is not the same as Git. Both are different. Git is the version control system, and GitHub is a cloud hosting service providing support for hosting your repositories online.

These are the main advantages of using GitHub.

1. You can host your entire repository online. That means if something unexpected happens and your system crashes, you can always clone your data from there.
2. An online remote can be forked by multiple users, and parallel development can be done. Using Pull Requests makes it really easy to merge the changes.
3. Built-in issue tracking. People can raise issues, and you can use the tracker to track each of them and solve them separately, or assign them within your team.
4. Opensource. You get to find a lot of interesting projects such as yours from which you can learn a lot. You can make your project open source, and other people who use your code can contribute back if they like.
5. Releases & Wiki support.
6. It will count as your personal portfolio when applying for tech jobs. Most people count it as experience too.

There are many other benefits too, such as integrations with CI build systems, built in dependency and vulnerability analysis, etc.,

What are the different DevOps servers available in the market? 

The best DevOps tools for 2021

We’ll start with the development stage of the DevOps cycle, then look at the operations stage.

It’s important to note that there can be overlaps between the functionality of these DevOps tools — sometimes you can perform the same action with different types of tools, and you can use them in several competing combinations in your DevOps stack.

Source code management and build tools

The DevOps lifecycle starts with creating, storing, analyzing, and reviewing your source code. Source code management includes tasks such as version control, issue tracking, code review, packaging, and more.

1. Git: Distributed source code management and version control system

Git is one of the most popular DevOps tools, widely used across the software industry. It’s a distributed SCM (source code management) tool, loved by remote teams and open-source contributors. Git allows you to track the progress of your development work. You can save different versions of your source code and return to a previous version when necessary. It’s also great for experimenting, as you can create separate branches and merge new features only when they’re ready to go.

By default, Git is a command line tool, but you can also download the GUI client that lets you manage your source code from a user-friendly graphical interface. Many source code editors such as Visual Studio Code also come with a built-in Git editor tool.

To integrate Git with your DevOps workflow, you also need to host your source code in repositories where your team members can push their work. Currently, the most popular online Git repo hosting services are Github, Gitlab, and Bitbucket. You can host both public and private repositories on these platforms, track and discuss issues, and manage releases. They also provide you with additional DevOps features, such as built-in continuous development and security functionalities, code review, auditing and collaboration tools, and more.

2. Jira: Issue and project tracking platform

Jira is a popular issue tracking and project management platform. It’s a product of Atlassian, a renowned IT company that develops several highly sought-after software tools, including Confluence, Bitbucket, Trello, Statuspage, and others. You can use Jira either as SaaS or on-premise.

As Jira started as a bug tracking tool, this is how most developers still think of it. Its user-friendly interface allows you to see the development status of your projects in context, create branches and pull requests, view commits, manage dependencies and releases, visualize progress, and more.

The project management features were added later to Jira and complement the bug tracking capabilities. As Jira focuses on Agile software development (i.e. methodologies that center around iterative development), it comes with Scrum and Kanban boards (two competing implementations of Agile), roadmaps, advanced reporting tools, and others.

Jira also has an advanced automation engine that enables you to build automation rules using a handy drag-and-drop interface. Not only can you automate Jira tasks, you can also connect to tools like Bitbucket, GitHub or Microsoft Teams, and include them in your automation workflows.

3. SonarQube: Automated code review tool

SonarQube is an open-source code review tool (see GitHub repo) that supports 27 programming languages. It’s ideal for analyzing the source code of applications written in multiple languages. It helps you keep the focus on code quality and security during the entire development process by automatically checking your code base against thousands of static code analysis rules.

Quality gates are SonarQube’s most notable feature. SonarQube scores your source code for bugs, vulnerabilities, code smells, coverage, and duplications, and assesses whether it passes the “quality gate” for public release. You can also use quality gates to analyze pull requests so you can quickly decide whether to merge them or not.

In addition to analyzing overall code health, SonarQube also highlights newly introduced issues. Plus, it provides you with useful visualizations that provide insight into the overall state of your code base. It works both in the cloud and on-premises and integrates with many DevOps tools, including GitHub, GitLab, Jenkins, Azure Pipelines, Bitbucket, and others. You can set SonarQube up with minimal configuration.

As SonarQube helps improve security by introducing continuous code inspection into your DevOps workflow, it’s also seen as one of the best DevSecOps tools.

4. Gradle: Multi-language build automation tool

Your DevOps stack needs a reliable build tool. Apache Ant and Maven dominated the market for years, but Gradle’s popularity has steadily grown since it showed up on the scene in 2009. Gradle is a versatile tool that supports multiple programming languages, including Java, C++, Python, and others. It’s also supported by popular IDEs (Integrated Development Environments) such as Netbeans, Eclipse, and IntelliJ IDEA. Plus, Google selected Gradle as the official build tool for Android Studio.

While Maven and Ant use XML for configuration, Gradle uses a Groovy-based DSL (domain-specific language) for describing builds. In addition to the default DSL, the Gradle team also released a Kotlin-based DSL to enable developers to write their build scripts in Kotlin. Gradle supports multiple repository formats including Maven-compatible artifact repositories, so dependency management will be familiar if you have prior experience with Maven. You can also import your Ant builds into Gradle.

Gradle allows you to use incremental builds that check if inputs and outputs have changed since the last run. This feature saves a significant amount of compile time — according to Gradle’s performance measurements, it’s up to 100 times faster than Maven. This performance gain is achieved partly by incrementality, and partly by Gradle’s build cache and daemon (background process). The build cache reuses task outputs, while the Gradle Daemon keeps build information hot in memory in-between builds.

All in all, Gradle allows faster shipping and comes with a lot of configuration options.

Containerization platforms

Containerization is a lightweight form of virtualization that packages up the source code of an application and all its dependencies as a container image, so that it can be deployed to different environments without further configuration.

5. Docker: Open-source containerization platform

Docker has been the number one container platform since its launch in 2013, and it continues to improve. It’s widely regarded as one of the most important DevOps tools. Docker has made containerization popular in the tech world, mainly because it makes distributed development possible and helps automate the deployment process. It isolates applications into separate containers, so they become portable across environments and more secure. Docker apps are OS- and platform-independent. You can use Docker containers instead of virtual machines such as VirtualBox.

With Docker, you don’t have to worry about dependency management. You can package all dependencies within the app’s container and ship the whole thing as an independent unit. Then, you can run the app on any machine or platform without a headache.

Docker integrates with Jenkins and Bamboo (see both below), too. Using it together with one of these CI/CD servers can further improve your DevOps workflow. Docker is also widely used in cloud computing, as all major cloud providers (including Amazon Web Services and the Google Cloud Platform) come with Docker support. So if you’re planning a cloud migration, Docker can ease the process for you.

6. Kubernetes: Automated container-orchestration platform

Kubernetes is a container orchestration platform that takes container management to the next level. It was founded by a couple of Google engineers who wanted to find a solution to manage containers at scale. Kubernetes works well with Docker or any of its alternatives to help you group your containers into logical units.

You may not need a container orchestration platform if you have only a few containers. However, it’s the next logical step once you reach a certain level of complexity and need to scale your resources. Kubernetes allows you to automate the process of managing hundreds or thousands of containers.

With Kubernetes, you don’t have to tie your containerized apps to a single machine. Instead, you can deploy to a cluster of computers, with Kubernetes automating the distribution and scheduling of containers across the whole cluster.

A Kubernetes cluster consists of one master and several worker nodes. The master node implements your pre-defined rules and deploys the containers to the worker nodes. Kubernetes monitors everything to prevent mishaps - for example, it will notice when a worker node is down and redistribute the containers.

CI/CD and deployment tools

CI/CD stands for continuous integration and continuous delivery/deployment. Continuous integration is a software development practice that merges the work of all developers working on the same project, while continuous delivery ensures reliable and frequent releases.

In DevOps culture, the two practices are combined to enable teams to ship software releases effectively, reliably, and frequently.

7. Jenkins: Open-source automated CI/CD server

Jenkins is one of the most popular DevOps automation tools in the market. It’s an open-source CI/CD server that allows you to automate the different stages of your delivery pipeline. The main reason for Jenkins’ popularity is its huge plugin ecosystem. With more than 1,800 plugins, it integrates with almost all DevOps tools, including Docker, Puppet, Octopus Deploy, and more.

With Jenkins, you can set up and customize your CI/CD pipeline according to your own needs. It’s easy to get started with Jenkins, as it runs natively on Windows, Mac OS X, and Linux. It’s easily installed with Docker. You can set up and configure your Jenkins server through a web interface. If you’re a first-time user, you can choose to install it using plugins or create your own custom configuration.

With Jenkins, you can iterate and deploy new code as quickly as possible. It also allows you to measure the success of each step of your pipeline. You can use it either as a simple CI server solely on the development side or as a complete CI/CD solution that also takes care of your deployment workflow.

8. Bamboo: Commercial automated CI/CD server

Bamboo is Atlassian’s CI/CD server solution, and has many similar features to Jenkins. Both are popular DevOps tools that allow you to automate your delivery pipeline from builds to deployment. However, while Jenkins is open-source, Bamboo comes with a price tag. So, is it worth choosing proprietary software over a free alternative? It depends on your budget and goals.

Bamboo has many pre-built functionalities that you have to set up manually in Jenkins. This is also the reason why Bamboo has fewer plugins (around 190 compared to Jenkins’ 1,800+). And actually, you don’t need a lot of plugins with Bamboo, as it does many things out of the box.

Bamboo seamlessly integrates with other Atlassian products such as Jira and Bitbucket. You also have access to built-in Git and Mercurial branching workflows and test environments. The bottom line is that Bamboo can save you a lot of configuration time. It also comes with a more intuitive UI with tooltips, auto-completion, and other handy features.

9. Octopus Deploy: Automated deployment and release management platform

Octopus Deploy is an advanced deployment and release management tool. It allows you to automate your deployment process across multiple environments and releases, and can be used together with CI/CD servers such as Jenkins or Bamboo to trigger an Octopus deployment after every build. Octopus then creates a new release and deploys it to different environments, including development, testing, staging, production, and other environments.

Use Octopus Deploy to package your application as a ZIP, .tar.gz, JAR/WAR, or NuGet file. You can define your own deployment process that Octopus will run automatically, but if you need some inspiration, Octopus has an extensive deploy library that lists almost 500 templates of frequent deployment workflows.

Octopus Deploy supports multiple cloud services, so you can easily deploy your application straight to the cloud. It also has a feature called ‘runbooks’ to automate common emergency operation tasks such as website failover and restoration. Runbooks include all the necessary permissions the different elements of your infrastructure require, so even our non-technical team members can run them.

A deployment management tool such as Octopus Deploy is recommended if you have a complex deployment workflow consisting of several environments (called ‘deployment targets’ by Octopus).

Configuration management tools

Configuration management tools speed up deployment by automating the maintenance, configuration, and orchestration processes of your entire infrastructure.

10. Puppet Enterprise: A platform-independent configuration management platform

Puppet Enterprise is a cross-platform configuration management platform. It allows you to manage your infrastructure as code. As it automates infrastructure management, you can deliver software faster and more securely. Puppet also provides developers with an open-source tool for smaller projects.

If you’re dealing with a larger infrastructure, you might find value in Puppet Enterprise’s extra features, such as:

• Real-time reports
• Role-based access control
• Node management

With Puppet Enterprise, you can manage multiple teams and thousands of resources. It automatically understands relationships within your infrastructure. It deals with dependencies and handles failures smartly. When it encounters a failed configuration, it skips all the dependent configurations as well. Puppet has more than 6,700 modules and integrates with many popular DevOps tools.

11. Ansible: A YAML-based configuration management tool

Ansible is an open-source configuration management tool sponsored by Red Hat. In functionality, it’s similar to DevOps automation tools on the deployment end of the CI/CD pipeline like Puppet (see above) and Chef. You can use it to configure your infrastructure and automate deployment. Compared to similar DevOps tools, its main selling points are simplicity and ease of use. Ansible follows the same Infrastructure-as-Code (IAC) approach as Puppet. However, it uses the super-simple YAML (Yet Another Markup Language) syntax. With Ansible, you can define tasks in YAML, while Puppet has its own declarative language.

Agentless architecture is another frequently mentioned feature of Ansible. As no daemons or agents run in the background, Ansible is a secure and lightweight solution for configuration management automation. Similar to Puppet, Ansible also has multiple modules.

If you want to better understand how Ansible fits into the DevOps workflow, take a look at this post by the Red Hat Blog, showing how to use Ansible for environment provisioning and application deployment within a Jenkins pipeline.

Cloud DevOps tools

These days, it’s possible to run your entire DevOps stack in the cloud, using the managed DevOps solutions of cloud service providers.

12. Azure DevOps Services: Microsoft’s cloud-based integrated DevOps platform

Azure DevOps (formerly known as Visual Studio Team System or VSTS) is Microsoft’s integrated DevOps platform. It enables you to manage your entire DevOps cycle from one integrated interface. While Azure DevOps Services is a cloud DevOps tool that you can use as a SaaS (Software-as-a-Service) application, Azure DevOps Server is the on-premise version of the same tool that you can self-host in your own data center.

Microsoft’s DevOps solution is a collection of tools, each addressing a different stage of your workflow. Azure Boards helps with planning and project management, Azure Pipeline is a CI/CD tool, Azure Repos includes cloud-hosted Git repos, Azure Test Plans is a testing toolkit, and Azure Artifacts let you create, manage, and deploy packages.

However, you don’t have to use all the tools included in Azure DevOps Services if you don’t want to, as you can also subscribe to them individually. If you need any additional features, you can also find more than 1,000 Azure DevOps extensions in the Visual Studio Marketplace, such as integrations, analytics, visualizations or crash reporting.

13. AWS DevOps: Amazon’s cloud-based DevOps toolkit

AWS DevOps is a part of Amazon Web Services and consists of a set of integrated DevOps tools that let you manage your entire software development lifecycle. Even though AWS is primarily used in the cloud, you can also run all the tools on-premise, with the help of AWS Outposts that let you install any element of the AWS infrastructure on your own in-house server.

Unlike Azure DevOps Services, which is a PaaS (Platform-as-a-Service), AWS is an IaaS (Infrastructure-as-a-Service) solution — so it’s tightly linked to the underlying infrastructure. While you can deploy packages from Azure DevOps Services to another infrastructure such as AWS, the reverse is isn’t possible. From AWS DevOps, you can only deploy to the AWS infrastructure like EC2 (Elastic Compute Cloud) or S3 (Simple Storage Service).

In the AWS DevOps toolkit, you can find a CI/CD service called AWS CodePipeline, a fully managed build tool called AWS CodeBuild, a deployment automation tool called AWS CodeDeploy, a DevOps project management platform called AWS CodeStar, and more. Overall, AWS DevOps is likely the best DevOps platform for current or future users of Amazon Web Services.

Monitoring and error reporting platforms

The last stage of the DevOps cycle is monitoring your application and infrastructure in real-time, so that you can react quickly when anything goes wrong.

14. Raygun: Error monitoring and crash reporting platform

Raygun is a full-stack performance and error monitoring solution that gives you actionable, real-time insights into the quality of your web and mobile applications. Its DevOps tools help you diagnose issues and track them back to the exact line of code, function, or API call. Raygun also comes with an APM tool for application performance monitoring that fits well with the error management workflow, automatically identifying your highest priority problems and creating issues for you.

Raygun APM can help you make the most out of other DevOps tools, as you are always notified about the problems. Since it automatically links errors back to the source code, Raygun brings Development and Operations together by providing one source of truth for the whole team.

15. Nagios: Infrastructure monitoring system

Nagios is one of the most popular free and open-source DevOps monitoring tools. It helps you monitor your infrastructure to find and fix problems. With Nagios, you can keep records of events, outages, and failures.

You can also keep an eye on trends using Nagios’ graphs and reports. This way, you can forecast outages and errors and detect security threats. Although there are many DevOps tools for infrastructure monitoring, Nagios stands out due to its rich plugin ecosystem.

Nagios offers four open-source monitoring solutions: Nagios Core is a command line tool, Nagios XI provides you with a web-based GUI (graphical user interface) and monitoring wizard, Nagios Log Server lets you search log data and set up alerts about potential threats, and Nagios Fusion allows you to monitor multiple networks at the same time.

As Nagios is a free tool, you’ll need to configure it by yourself. Alternatively, you can opt for one of their support plans to access to the help and expertise of the Nagios team.

Which DevOps tools are right for your team?

Finding the best DevOps tools for your specific requirements takes some testing and experimentation. Open-source tools tend to take more time to set up and configure, while most commercial DevOps tools come with free trials that allow you to test and evaluate them at no cost. Ultimately, the decision comes down to what you’re trying to achieve and the resources you have available. While the process of selecting and implementing your toolkit might require some time up front, once you have it in place, you’ll reap the benefits of a more cohesive development workflow.

What are the drawbacks in waterfall software development model? 

The Waterfall model is an approach for developing software that breaks a project into finite phases. One should move to the next phase only when its preceding phase is reviewed and verified.

In the waterfall model, phases do not overlap.

=> Read more about the Waterfall Model here.

Figure 1 demonstrates the Waterfall model:

Advantages of the Waterfall Model:

• Simple and easy to understand and use.
• Rigid model – Each phase has specific deliverables and review processes.
• Documentation and artefacts meticulously maintained.
• Suitable for projects where requirements are well understood.

Disadvantages of the Waterfall model:

• Not suitable for projects where requirements are at a risk of changing.
• Cost of fixing defects is very high when detected at a later stage.
• Not a good model for complex and long projects.
• No working software is produced until late during the lifecycle.

Agile Model

Wikipedia defines the Agile Model as “a group of software development methods based on iterative and incremental development, where requirements and solutions evolve through collaboration between self-organizing, cross-functional teams.”

The model has its own principles that tend to bring the processes to the backseat.

=> Read more articles on Agile methodology here.

(Click on image to view enlarged)

Advantages of the Agile model:

• Customer involvement in the process.
• High ROI as working software is delivered frequently.
• Even late changes in requirements can be easily accommodated.
• Continuous improvement to both product and process.

Disadvantages of the Agile Model:

• Lack of emphasis on designing and documentation.
• The team should be stable and skilled.

How does agile methodology overcome the difficulties in waterfall model? 

Waterfall vs. Agile

The waterfall development model uses an approach that relies on a defined sequence of accomplishments to derive an endpoint. As the name implies, a development product’s life cycle flows steadily towards completion, much like how water travels down a waterfall.

The waterfall development method has its roots in industries, ranging from construction to assembly lines, in which sequential processes are the key to success. Waterfall methodologies also took hold in the early days of programming, where changes during development projects were costly or impractical. Applying a waterfall methodology to a programming project meant that all the requirements gathering and design work was done before any coding took place.

Although the ideology behind waterfall isn’t old, it was first described in the software development realm by Winston Walker Royce, who wrote a paper in 1970 explaining the methodology applied to software development projects. Waterfall has been the leading methodology for software development projects since its introduction and in most cases, adheres to the seven-step process outlined in Royce’s 1970 publication.

The Agile development methodology eschews a linear, sequence-based approach, and uses an incremental, iterative approach. Agile-based projects do not start with extensive planning and design, which means changing requirements can be incorporated into the development process. Cross-functional teams become an important piece of the Agile method puzzle. They incorporate planners, designers, developers and testers working in concert to create success iterations of the product. Those iterations are completed over fixed time periods (referred to as timeboxes) with the goal of creating a working product, which can be demonstrated to stakeholders for feedback, comments and change requests to introduce in future iterations.

 

Why Agile methodologies work.

To weigh the differences between waterfall and Agile methodologies, it is critically important to know how their characteristics can be interpreted based upon an individual’s point of view. For example, a software developer may find the idea of flexibility and reduced test cycles a pro, while a project manager may consider those two elements a con.

The most common pain points of waterfall methodologies include: long waits for releases, inability to make changes throughout the development process, and having to completely restart the development process due to changes in needs. Fortunately, these can be resolved with Agile methodologies.

The driving forces behind Agile methodology adoption.

Agile methodologies are becoming the go-to ideology to embrace productivity while allowing businesses to react to changes, market demands, and shifting priorities.

Agile methods offer more flexibility than waterfall derived projects. Agile is also customer facing, meaning that customer requests can be implemented on the fly, allowing the creation of software that adapts as needed and can change with each iteration to meet new challenges. For example, if a company using waterfall methodologies is developing software with a green background, then decides they need a blue background, this would be a major delay for the project. Using an Agile methodology, this would be a small change seamlessly integrated into development. The Agile methodology offers several perceived benefits, which is why the adoption of it as a development practice is growing. The benefits include:

• Reduced Waste: Continuous improvement proves to be a faster process, reducing waste in the form of time spent on design, redesign and addressing change.
• Increased Speed: Agile’s ability to bring forth rapid change, especially in the form of course corrections, means that deliverables can be constructed quicker.
• Improved Productivity: Agile is a more effective way for teams to work together, meaning that productivity will increase due to improvements in communications, as well as insight quickly garnered by cooperative goal setting.
• Enriched Decision Making: Agile teams share wisdom and help to drive faster decisions based upon needs and not original design goals.
• Improved Confidence: With better decisions and a proactive approach, Agile improves the confidence of an organization. It makes it easier for the product to pivot, which allows the organization to react better to changing market conditions.
• Improved Trust and Safety: Agile has a different approach toward failure: it emphasizes learning from mistakes and leverages a blameless post-mortem, which improves the attitude towards failures. Ultimately, learning from failures and eliminating blame leads to improved trust among team members.

An HP online survey of 601 development and IT professionals shows that 51 percent of respondents are leaning towards Agile, while another 16 percent are already using Agile as their sole development method. Only 2 percent surveyed were pure waterfall environments, while only 7 percent were leaning towards waterfall. This is a clear indication that Agile is taking hold and is quickly becoming the new normal.

The path to Agile.

Adopting an Agile ideology is more than just claiming teams are Agile. Managers and decision makers should understand the components of what makes up an Agile ideology, especially when Scrum is the target ideology. One of the biggest struggles businesses encounter when shifting to Agile is the cultural shift. While having an Agile evangelist is an effective first step, the need to permeate the culture requires several major shifts.

• Goals shift from making money to customer satisfaction.
• Structure shifts from management overseeing to management enabling self-organized teams to contribute all they are capable of.
• Workflow shifts from bureaucratic to cyclical with direct customer feedback.
• Values shift from efficiency and predictability to transparency and continuous improvement.
• Communication shifts from hierarchical to lateral.

Scrum not only outlines the process, but also defines the roles of those involved. 

What are the different roles in an agile team? 

A Scrum team may consist of:

• Product Owner: Shares the vision of the product, prioritizes the functionalities to be built, and makes key decisions on behalf of the team. The Product Owner is also responsible for maintaining the product backlog, bridging the gap between the developers and other stakeholders, managing the end-user (or customer) expectations, and managing the budget.
• Scrum Master: Primarily responsible for solving problems team members may be facing while building the product. The Scrum Master does not have to completely understand all requirements, but must still be able to find solutions to situations. Ultimately, the Scrum Master is responsible for creating the best possible working conditions for team members so they can meet the goals of each sprint effectively.
• Scrum Development Team: A cross-functional team that is responsible for developing the product. The team consists of developers, business analysts, testers and other liaisons. The goal of the team is to work together and in tandem while building the application. Team activities are aligned with the targets associated with the specific sprint. Team members must identify the complexity of assigned tasks and allocate the appropriate resources. Team members are expected to communicate the status of project and issues they are facing to Scrum Masters on a daily basis. Team members also are responsible for demonstrating tasks completed to product owners during sprint reviews.

What are the different ceremonies in an agile team?

At a glance:

• The four agile ceremonies are Sprint Planning, Daily Stand-Up, Sprint Review and Sprint Retrospective
• Ceremonies in agile facilitate visibility, transparency, and collaboration.
• Each ceremony has a clear structure and objective.
• Clear communication, flexibility, and cultural alignment are the keys to successful ceremonies.

What are the main agile ceremonies?

Agile ceremonies refer to the four events that occur during a Scrum sprint. Other forms of agile development, such as Kanban and Lean, also have similar practices.

The agile ceremonies list includes:

1. Sprint Planning
2. Daily Stand-Up
3. Sprint Review
4. Sprint Retrospective

While each ceremony is different, they facilitate the same overall purpose. The ceremonies bring teams together with a common goal under a regular rhythm, and they help teams get things done.

"With today's enterprises under increased pressure to respond quickly to the needs of their customers and stakeholders, they must bring new products to market faster and accelerate improvements to existing solutions and services." - State of Agile Report

Why are agile ceremonies important?

Agile ceremonies help organizations adapt to change and succeed. With work planned in smaller portions and over shorter timeframes, they help teams quickly shift direction and course-correct when needed. They form a key part of the broader agile approach that’s now widely adopted in organizations worldwide.

With agile ceremonies, teams in your organization can benefit from:

• Enhanced ability to manage changing priorities
• Acceleration of software development
• Increase in team productivity
• Improved business and IT alignment

It’s important to remember that while ceremonies are an essential part of Scrum, they’re just one of many rituals that help create agile teams and workplaces. To realize the true benefits of agile, you’ll need to do more than include one or more of the ceremonies into your waterfall project.

1. Sprint Planning

The Sprint Planning ceremony sets teams up for success by ensuring everyone understands the sprint goals and how to achieve them.

Structure	Attendees	Timing	Duration	Agile Framework
The Product Owner brings the product backlog to discuss with the Development Team. The Scrum Master facilitates. Together, the Scrum Team does effort or story point estimations. The product backlog must contain all the details necessary for estimation. The Product Owner should be able to clarify any doubts regarding the product backlog.	The entire Scrum Team (the Development Team, Scrum Master, and Product Owner)	At the beginning of each sprint	One to two hours per week of iteration. So, if you're planning a two-week sprint, your Sprint Planning should last two to four hours.	Scrum. Although Kanban teams also plan, they do it less formally and per milestone, not iteration.

Outcomes

After some team negotiation and discussion, you should have a clear decision on the work that the Development Team can complete during the sprint by the end of Sprint Planning. This is known as the sprint goal.

The sprint goal is an increment of complete work, and everyone should feel confident about the commitment.

The product backlog defines priorities that affect the order of work. Then, the Scrum Master transforms that decision into the sprint backlog.

Top tips

• Focus on collaboration rather than competition.
• Break user stories into tasks to get things more operational for the Development Team. If there's time, assign those tasks during the event.
• Factor in public holidays and any team member’s time off or vacations.
• Keep your team’s pace in mind – a track record of the time it took to implement similar user stories would be helpful.
• Focus on the product backlog and nothing else in terms of work for the sprint.

Easy Agile User Story Maps helps Agile teams:

• Clearly visualize team capacity
• Prioritize user stories by business value
• Quickly drag new stories into the story map
  

2. Daily Stand-Up

The daily stand-up brings the team together and sets everyone up for the day. The team uses this time to identify blockers and share plans for the day.

Structure	Attendees	Timing	Duration	Agile framework
This is an informal, standing meeting. All members of the Development Team inform everyone about what they did the day before and what they’re doing today. Members discuss any blockages they have and ask for help from the team if required. Due to time restrictions, the updates should be brief.	Development Team, Scrum Master, Product Owner (optional)	Daily, usually in the morning	Short and sharp. No longer than 15 minutes	Scrum and Kanban

Outcomes

The Scrum Master should clear all the blockages that slow down or prevent the Development Team from delivering. As a result, the development process might need to change.

This daily pulse check keeps the team in sync and helps build trust. Together, the group finds ways to support and help each other.

Top tips

• Use a timer to keep this meeting to 15 minutes.
• Hold your stand-up at the same time every day.
• Only discuss the work for the day ahead.
• If the team is distributed, use video conferencing with cameras on.
• Long discussions should happen after the event.
• As the stand-up encourages progress, everyone should provide an update, and everyone should feel accountable.

3. Sprint Review

The Sprint Review is the time to showcase the team’s completed work and gather feedback from stakeholders. A variety of attendees from outside the team offer valuable insights from different viewpoints. This event also helps build trust with both external and internal stakeholders.

Structure	Attendees	Timing	Duration	Agile framework
The Scrum Master takes on the logistics of event preparation. The Product Owner should ask stakeholders questions to gather as much feedback as possible. They should also answer any of their stakeholder’s questions.	Development Team, Scrum Master, Product Owner. Optionally, management, customers, developers, and other stakeholders	At the end of the sprint	One hour per week of the sprint. In a one-week sprint, the Sprint Review lasts one hour.	Scrum and Kanban. Kanban teams do these reviews after the team milestones, not sprints.

Outcomes

After this ceremony, the Product Owner might need to adjust or add to the product backlog. They might also release product functionality if it's already complete.

Top tips

• Schedule in time to rehearse before the meeting to help your team present with confidence, especially if external stakeholders are coming along.
• Don’t showcase incomplete work. Review your Sprint Planning and the original criteria if you’re not sure whether the work is complete.
• Besides product functionality, focus on user experience, customer value, and the delivered business value.
• Consider ways you can introduce a celebratory feel to acknowledge the team’s effort.

4. Sprint Retrospective

In this final scrum ceremony in the sequence, you look back on the work you’ve just done and identify ways to do things better next time. The Sprint Retrospective is a tool for risk mitigation in future sprints.

Structure	Attendees	Timing	Duration	Agile framework
The teams discuss what went well throughout the sprint and what went wrong. The Scrum Master should encourage the Development Team to speak up and share not only facts but also their feelings. The goal is to gather rapid feedback for continuous improvement in terms of process. It’s also an opportunity to emphasize good practices that the team adopted and should repeat.	Development Team, Scrum Master, Product Owner (optional)	At the end of the sprint	45 minutes per sprint week	Scrum and Kanban (occasionally)

Outcomes

After this session, the team should clearly understand the problems and the wins that happened throughout the iteration. Together, the group comes up with solutions and an action plan to prevent and identify process problems in the next sprint.

Top tips

• Focus on both facts and feelings
• Gather information that helps you focus on continuous improvement – this might include tools and relationships
• Be honest and encourage ideas that solve process-related problems
• Even if everything went well, have this meeting – retrospectives provide ongoing guidance for the next sprint.

"With the speed of change expected to continue, the need has never been greater for an operating model that keep up." - McKinsey

Agile lessons to live by

As a team of experienced agile practitioners, we’ve picked up some key learnings about what it takes to get the most out of your agile ceremonies and create the foundations of a truly agile organization.

Here are our top tips to make your ceremonies a success:

• Be deliberately present - During the ceremonies, remember to take moments to pause and remind yourself of why you’re there. Show others that you’re present by giving them full attention and using your body language. In a remote setting, angle  your camera as though you’re sitting across from them, look into the lens regularly, and use a distraction-free background.
• Practice active listening - Think about what the person is saying, who what they are, and what they needwant from you. Are they looking for a soundboard, do they need your help or opinion, or are they looking for an emotional connection?
• Understand motives - Understand the motivations of your teammates before speaking. Consider why they should care about what you’re saying by connecting your message with their own motivations. Provide context where possible to let them know why your message matters.
• Be flexible - It's important to remember that there is not a one size fits all approach to agile ways of working. What works for one team may not work for another, so you need to experiment to find out what works then tailor processes to suit your team's needs.
• Create cultural alignment - The best processes in the world won’t deliver what you need if you don’t have the culture to support their delivery. Agile ceremonies need to be supported by a culture where people are actively engaged, confident to raise issues, and value continuous improvement.

Agile ceremonies lead to better results

While it can take time for teams new to agile to adjust to agile ceremonies, they are worth the effort. By providing a clear structure and achievable outcomes, they help align everyone on the product, communication, and priorities.

The result? Agile teams that provide better quality products faster – and deliver real business outcomes.

Wherever your organization is on your agile journey, it’s worth keeping in mind that each team and each suite of products are different, so there’s no standard recipe for success. The good news is that by working within the continuous improvement mindset the agile framework promotes; you too can iterate and improve your agile ceremonies over time.