Achieving Safety and Velocity in CI/CD

Achieving Safety and Velocity in CI/CD with Fast Feedback Loops

In the fast-paced world of software engineering, the ability to quickly iterate and adapt is crucial to success. The utilization of agile methodologies is well-known to put a development team at a significant advantage. One way this can be achieved is the implementation of fast feedback loops.

What are Fast Feedback Loops?

Fast feedback loops are repeating cycles where team members can share information, evaluate, revise, and then repeat the process iteratively. This approach ensures that feedback is continuously integrated into the development process, leading to more refined and effective outcomes.  

One non-engineering example of a fast feedback loop would be the iterative process of writing this blog post, which consists of:  

• creating an outline

• requesting feedback

• writing a draft

• requesting additional feedback

• submitting a final draft for editing

Each step involves receiving and incorporating feedback to improve the final product. If the outline was skipped and feedback was never solicited before the final draft was written, any requested edits could have resulted in a full rewrite. This would needlessly waste the time of both the writer and editor. In addition, the content of the blog post may have been less refined, which would waste the time of the reader.

Core Principles of Fast Feedback Loops

Here are some of the core principles that underpin fast feedback loops, each contributing to achieving the goal of continuous improvement and increased velocity.

1. Daily standup meetings: These meetings empower developers to ask questions about their assigned stories, with minimal wait time required to receive answers. This immediate feedback helps keep the development process on track and addresses any issues promptly.
2. Sprint reviews: Sprint reviews provide an opportunity for stakeholders to make changes early in the development cycle, while it is still inexpensive to do so. This early intervention can save time and resources by ensuring that the project is aligned with stakeholder expectations from the outset – a la “build it just one time.”
3. Automated build pipelines: Automated build pipelines reduce the amount of DevOps toil involved in building and deploying releases. By automating these processes, teams can focus more on development and less on the mechanics of deployment.
4. Automated testing: Configuring your deployment pipeline to run a set of automated tests and reviewing these testing metrics at a regular cadence ensures complete coverage. This provides safety and peace of mind during deployments, as it helps catch issues early and ensures that the codebase remains stable.
    

Fast Feedback Loop Usage within WellSky Enterprise

‍
WellSky Enterprise teams have implemented several strategies to leverage fast feedback loops effectively:

• Message audit insights via the Interop Hub Console: This tool provides a user-friendly interface that developers and solutions teammates can access to review recent transactions. It allows for a comprehensive assessment of the current version deployed to any given environment, facilitating quick feedback and adjustments.
  

• Implementation of Open Telemetry: Open Telemetry reduces triage time with detailed tracing and forwards aggregate data to our Application Performance Monitoring tool, providing observability to stakeholders. This detailed insight helps in quickly identifying and resolving issues, ensuring smooth operation.
  

• Automated deployment to Stable/QA environments: Our pipelines are configured to deploy to Stable/QA environments with every merge to the main branch. This maximizes the amount of time that changes are available for testing—both directly by QA analysts and passively when consumed by other company solutions—before the changes are scheduled to be released to production. This new process yields numerous benefits:
  • The building and deployment of the release version is no longer a manual process requiring human effort.
  • If an issue exists in the release, there is a higher likelihood of it being detected while there is still sufficient time for remediation. This minimizes the risk of features being removed from the release, the release being delayed, or deployed and then rolled back.
  • Developers should always self-test their deployed changes after merging, but in the event that an issue is caught during QA, it may not always be clear which change is to blame if the release includes several bundled changes. Conversely, when we automatically deploy each merge to Stable/QA, and something breaks, we know immediately what code needs to be revisited and which developer the bug should be assigned to.
  • Rolling back the offending commit in Stable/QA is often a simple operation that can be performed from Azure DevOps. This alleviates the need to revert the commit in the repository and is therefore known as “rolling forward.”

   
Looping it All Together

Fast feedback loops are an essential component of modern software development. By integrating continuous feedback into the development process, teams can ensure that they are always moving toward a more refined and effective product. The implementation of fast feedback loops within WellSky Enterprise has enhanced our ability to deliver high-quality software, more rapidly, and with less risk. Embracing these principles can help any team achieve similar success.