C Get Log From Continuous Process

Speed is the key to modern software development. The monolithic all-or-nothing paradigm of traditional waterfall software development has largely been replaced by rapid iterative techniques that support development and release. These techniques go by several names, including continuous integration, continuous delivery and continuous deployment.

While each technique offers slight differences, the common emphasis on continuous iteration has changed the nature and power of software development. Businesses can get software to market faster, test innovative new features or architectures while minimizing risk and cost, and effectively refine products over time.

Such iteration relies on well-planned and active pipelines designed to support multiple iterations in various stages of the development cycle simultaneously -- and keep entire development teams constantly busy. As one build is pushed to deployment, the next build undergoes testing, while the very latest build in the cycle is coded.

Let's take a closer look at these continuous approaches, see how each works, weigh the tradeoffs and consider best practices.

What do CI and CD mean?

The close relationship between continuous integration, continuous delivery and continuous deployment can sometimes be confusing, especially when combined in the cyclical process known as CI/CD. It's important to understand the differences between each approach.

Continuous integration (CI) focuses on the early stages of a software development pipeline where the code is built and undergoes initial testing. Multiple developers work on the same codebase simultaneously and make frequent commits to the code repository. Build frequency can be daily or even several times per day at some points in the project's lifecycle. These small, frequent builds enable easy and low-risk experimentation, as well as the ability to easily roll back or abandon undesirable outcomes.

CI employs a variety of tools and automation techniques to create builds and shepherd them through initial testing, such as sniff or unit testing, along with more comprehensive integration testing. CI is also noted for its rapid and detailed feedback. The limited nature of each iteration means that bugs are identified, located, reported and corrected with relative ease.

Ultimately, CI ends when a build successfully completes initial testing and is ready to move to more comprehensive testing. Preparation might include packaging the build into a deployable image, such as a container or virtual machine (VM) image, before making it available to dedicated testers.

Continuous delivery (CD) picks up where CI leaves off. It focuses on the later stages of a pipeline, where a completed build is thoroughly tested, validated and delivered for deployment. Continuous delivery can -- but does not necessarily -- deploy a successfully tested and validated build.

CD likewise relies heavily on tools and automation to take a build through advanced testing, including functional, user acceptance, configuration and load testing. These validate that the build meets requirements and is ready for use in a production environment. Again, small incremental iterations ensure that any problems revealed in testing are identified and remediated quickly and less expensively than traditional software development approaches.

Continuous deployment (also CD) follows the same basic steps as continuous delivery. The principal difference between delivery and deployment is that continuous deployment automatically deploys each validated build to production. By comparison, continuous delivery typically just stages the validated build for manual deployment or other human authorization.

How continuous integration and continuous delivery fit together.

Continuous deployment further accelerates the iterative software development process by eliminating the lag between build validation and deployment. However, such a paradigm could also allow undetected flaws or vulnerabilities to slip through testing and wind up in production. For many organizations, automated deployment presents too many potential risks to enterprise security and compliance. These teams prefer the continuous delivery paradigm in which humans review a validated build before it is released.

The common theme through these three continuous paradigms is a heavy reliance on automation and testing to drive the process. Ideally, a developer need only "press the button" to whisk a new build from the code repository through testing and on to delivery or deployment. This tantalizing proposition depends on the quality of the testing, the integrity of the automation behind it, and the careful attention of testers and software engineers.

Benefits and challenges of a CI/CD pipeline

In a CI/CD pipeline, everything is designed to happen simultaneously: Some software iterations are being coded, other iterations are being tested and others are heading for deployment. Still, CI/CD involves important tradeoffs between benefits and drawbacks.

The benefits of a CI/CD pipeline include the following:

• Efficient software development. Smaller iterations (steps) allow for easier and more efficient testing. The limited scope of code in each new iteration, as well as the scope to test it, makes it easier to find and fix bugs. Features are more readily evaluated for usefulness and user acceptance, and less useful features are easily adjusted or even abandoned before further development is wasted.
• Competitive software products. Traditional software development approaches can take months or years, and formalized specifications and requirements aren't well suited to changing user needs and expectations. CI/CD development readily adapts to new and changing requirements, which enables developers to implement changes in subsequent iterations. Products developed with CI/CD can reach market faster and with more success.
• Freedom to fail. CI/CD's rapid cyclicality enables developers to experiment with innovative coding styles and algorithms with far less risk than traditional software development paradigms. If an experiment doesn't work out, it probably won't ever see production and can be undone in the next rapid iteration. The potential for competitive innovation is a powerful driver for organizations to use CI/CD.
• Better software maintenance. Bugs can take weeks or months to fix in traditional software development, but the constant flow of a CI/CD pipeline makes it easier to address and fix bugs faster and with better confidence. The product is more stable and reliable over time.
• Better operations support. Regular software releases keep operations staff in tune with the software's requirements and monitoring needs. Administrators are better able to deploy software updates and handle rollbacks with fewer deployment errors and needless troubleshooting. Similarly, IT automation technologies can help speed deployments while reducing setup or configuration errors.

Despite these compelling benefits, business leaders and development teams must consider some of the potential pitfalls of CI/CD pipelines:

• Dedication to automation. CI/CD relies on the consistency of an established tool set and strong automation to build, test and deploy each build. This demands a serious intellectual investment to implement and manage the automation, which can involve a steep learning curve. Changes to the development process or tool set can profoundly impact the CI/CD pipeline, so CI/CD is often employed in mature and active development environments.
• Staff discipline and planning. A CI/CD process cannot bring full value to the business if it's not constantly generating new builds, testing release candidates and deploying selected candidates to production. This requires careful planning and expert project management skills. Developers must adhere to established development guidelines to ensure quality, style and architectural standards. Meanwhile, business leaders and project stakeholders can be extremely uncomfortable with automated deployments in continuous deployment paradigms, and meetings for manual go/no-go deployment decisions can be fraught with stress over unknown or unforeseen consequences. This can cause unnecessary delays -- all while new builds are coming through the pipeline.
• Communication and collaboration. No amount of automation and tooling is a substitute for good communication and collaboration among developers and project stakeholders. These interactions facilitate the rapid, efficient experimentation that makes CI/CD so powerful. Automation and tools are just means to that end.

Attributes of a good CD/CD pipeline

So what does effective CI/CD actually look like? Here are some common characteristics of a good CI/CD process:

• Speed. A CI/CD pipeline can have numerous steps and parts, but a build should be able to move through the pipeline (integration, testing, delivery and even deployment) in short order -- perhaps just minutes to complete an integration, and a few hours for testing cycles to finish. If it takes days to move a build through the pipeline, a great deal of valuable time is probably being wasted, and the process needs fine-tuning.
• Consistency. The processes and components used in one place or cycle are exactly the same in all places and cycles. For example, if a build results in a Docker container, that container is the object that's tested and moved through the pipeline to delivery or deployment. Developers can write scripts and create automation processes with confidence that such efforts will be successful and deliver the desired results every time. Processes that introduce variations or manual steps slow the pipeline and invite errors and inefficiency.
• Tight version control. Well-documented repositories or components and builds allow rapid restoration of previous builds when new builds or deployments go awry. Good version control facilitates fast, accurate and confident rollbacks -- perhaps to the previous working version -- whenever the need arises.
• Automation. Extensive automation moves new code through integration, testing and delivery or deployment with little, if any, manual interaction. Ideally, a human does little more than develop and commit code, then wait for an approved pull request and approval before a deployment, if needed. A pipeline that depends on manual steps and processes is slow and prone to errors.
• Integrated feedback loops. A CI/CD pipeline is a loop that yields countless iterative steps to a completed project -- and each phase also offers a loop back to the beginning. A problem with the source code won't generate a build. A problem with the build won't move into testing. A problem in testing or after deployment will demand source fixes. The sooner a problem is identified, the faster, easier and cheaper it is to fix, keeping the overall pipeline in motion.
• Security throughout. Oversights and mistakes in programming and testing can create vulnerabilities and expose software to malicious activity. Thus, it is critical to infuse security best practices throughout the CI/CD pipeline. Tools such as vulnerability checkers can help spot potential security flaws in the code flowing through the pipeline, while additional security evaluations should take place during the testing phase.

The continuous delivery process involves several stages of checks, gates and feedback loops before final test acceptance and push to production.

What are the stages of a CI/CD pipeline?

The CI/CD pipeline combines continuous integration, delivery and deployment into four major phases: source, build, test and deploy. Each phase uses highly detailed processes, standards, tools and automation. Just as physical products made in a factory may benefit from customized manufacturing machines, software pipelines are frequently tailored to suit the specific needs of the project and the business.

Source. The first phase in a CI/CD pipeline is the creation of source code, where developers translate requirements into functional algorithms, behaviors and features. The tools employed for this depend on whether the development team is working in Java, .NET, C#, PHP or countless other development languages. Integrated development environments (IDEs) are often selected because they support a specific language in addition to various code-checking features, such as basic error detection, vulnerability scanning and adherence to established coding standards. Other source code and pipeline support tools, including code repositories and version control systems such as Git, typically form the foundation for building and testing phases.

There is no single pipeline for source creation. A business may have multiple source portions of the CI/CD pipeline depending on the various projects in development -- such as a server-side platform using C++, website applications using Java and mobile applications using Go. Similarly, coding features may vary between IDEs and projects due to different standards or vulnerabilities between projects, such as enterprise production systems versus a consumer app.

Build. The build process draws source code from a repository, establishes links to relevant libraries, modules and dependencies, and compiles (builds) all these components into an executable (.exe) file. Tools used in this stage also generate logs of the process, denote errors to investigate and correct, and notify developers that the build is completed.

As with source code creation, build tools typically depend on the selected programming language. A development group may employ independent tools to generate a build; many IDEs incorporate such build capabilities, which means they effectively support both the source creation and building phases within a CI/CD pipeline.

A build phase may employ additional tooling, such as scripts, to translate the executable file into a packaged or deployable execution environment, such as a VM (complete with an operating system and related components) or a container, such as Docker container with libraries and dependencies.

Test. While source code has already completed some static testing, the completed build now enters the next CI/CD phase of comprehensive dynamic testing. This usually starts with basic functional or unit testing to verify that new features and functions work as intended, and regression testing to ensure that new changes or additions do not accidentally break any previously working features. The build also undergoes a battery of tests for integration, user acceptance and performance. If errors occur during testing, the results are looped back to developers for analysis and remediation in subsequent builds.

Automation is particularly critical in the CI/CD test phase, where a build is subjected to an enormous array of tests and test cases to validate its operation. Human testing is typically too slow and subject to errors and oversights to ensure reliable or objective testing outcomes. Test specialists create comprehensive test cases and criteria but depend on test tools to implement testing and validation in a busy pipeline.

Deploy. Ultimately, the build passes the testing phase and is considered a candidate for deployment in a production environment. In a continuous delivery pipeline, it is sent to human stakeholders, approved and then deployed. In a continuous deployment pipeline, the build automatically deploys as soon as it passes its test suite.

The deployment step typically involves creating a deployment environment -- for example, provisioning resources and services within the data center -- and moving the build to its deployment target, such as a server. These steps are typically automated with scripts or through workflows in automation tools. Deployments also usually connect to error reporting and ticketing tools to find unexpected errors after the build is deployed and alert developers. Users can also submit bug tickets to denote real or perceived errors with the release.

Even the most wildly optimistic deployment candidates are rarely committed to production without reservation. Deployments frequently involve additional precautions and live testing periods, including beta tests, A/B tests, blue/green tests and other crossover periods to curtail or roll back unforeseen software problems and minimize the business impact.

What is an example of a CI/CD pipeline?

There is no single right way to build a CI/CD pipeline. Every pipeline can use different tools and include variations or alternate pathways to support project types with different scopes and sophistication. Regardless of the approach, an ideal CI/CD pipeline should meet three fundamental goals:

• Improve the software's quality.
• Make software development faster and more agile.
• Boost confidence in software deployment to production.

Let's examine a typical CI/CD pipeline, consider the activities within each stage and note several possible tools to tackle them.

Source

Developers write code using editors or IDEs. A development team may employ several editors or IDEs to support multiple languages for different projects.

Tools: Examples of software IDEs include Atom, Cloud9 IDE, Microsoft Visual C++ or Visual Studio, PyCharm and Xcode.

The source code is typically stored in a common shared repository, or repo, where multiple developers can access and work on the codebase at the same time. Repos also generally hold other parts of the software development process, such as artifacts (of compilation and linking), libraries, executables, modules, test scripts and suites. Repos provide a comprehensive version control system, which ensures developers work on the latest codebase and integrate the latest components in the build process.

Tools: GitHub, based on Git, is a popular repo; other examples of repositories include GitLab, Cloudsmith Package, Docker Hub for container projects, JFrog Artifactory, NuGet and SVN.

Once a developer commits changes to the codebase, those changes are saved to the version control system in the repository, which automatically triggers a new build.

Build

The build process typically involves multiple steps: Fetch the source code components from the repo, compile code, and link libraries and other modules. The result is a simple executable file, or a more complex assembly such as a deployable container or VM. All of the artifacts involved in the build process are typically retained in the repository. If there are problems or errors in the build, the process stops and issues are reported back to the developers for remediation. Typical problems include functional errors, such as a divide-by-zero math error, or missing components -- for example, a required library or module is not present in the build manifest.

Tools: Many IDEs include build tools natively tailored to the selected programming language. Alternatively, standalone build tools include Ant, Gradle, Make, Maven, Meister, Phing and Rake. Jenkins is a popular CI engine, but there are Jenkins alternatives such as Atlassian Bamboo, AWS CodePipeline, CircleCI, GitHub Actions and TeamCity.

The build stage may also include some basic testing for vulnerabilities, such as software composition analysis (SCA) and static application security testing (SAST).

Tools: Examples of SAST tools include Arctic Wolf Vulnerability Assessment, Fortify Static Code Analyzer and Netsparker. Vendors with SCA tools include Checkmarx, Kiuwan, Snyk, Synopsys and Veracode.

If the build completes successfully and passes initial test scans, it moves to the CI/CD testing phase.

Test

While some testing is indigenous to the build process, most happens after the build is successfully completed. This complex phase involves numerous steps and goals, including the following:

• Unit testing validates new features and functions added to the build.
• Dynamic application security testing (DAST) scans the build for security flaws, such as weak passwords or missing credentials.
• Interactive application security testing (IAST) analyzes traffic and execution flow to detect security issues, including those in third-party or open source components.
• Regression testing verifies that changes or additions do not harm previous features.
• Integration testing ensures the build operates with other applications or services.
• User acceptance testing assesses whether users are able to use new features and functions as intended.
• Performance testing ensures the build operates as required under load.

Developers and software testing specialists create test conditions that provide input to the build and compare the actual response or output to the expected response. If they match, the test is considered successful and the build moves on to the next test. If they do not match, the deviation is noted, and error information is sent back to the development team for investigation and remediation.

Tools: There are hundreds of test tools applicable through these tests. Each supports some level of automation, and some can perform multiple types of tests. Popular testing tools include Appian, Katalon, Kobiton, Kualitee, Micro Focus, Qaprosoft, Sauce Labs, Selenium, Telerik Test Studio and TestArchitect.

Not all builds that successfully complete the testing phase move into the deployment phase. Some builds may simply represent interim steps that need validation but are not yet ready for deployment. For example, developers may test an incomplete feature subset, flesh out the remaining feature subset in a subsequent build and then deploy it in its entirety.

Deploy

A build that successfully passes testing may be initially deployed to a test server; this is sometimes called a test deployment or pre-production deployment. A script copies a build artifact from the repo to a desired test server, then sets up dependencies and paths.

Once on a test server, the build can be configured to simulate a production environment; for instance, access to test databases and other applications may be enabled for "real-world" functional and performance evaluations. Much of this relies on automation but may involve human testing to shake down nuances of the build. This is sometimes called an alpha or development release, and involves only a small base of well-informed testers and users.

Pre-production deployment is typically the endpoint for continuous delivery pipelines. Once the build is completely validated and stakeholders have confidence in the build's stability and integrity, it can be deployed to an actual production environment. In a continuous deployment pipeline, once the build passes pre-deployment testing, it is automatically deployed to production.

To improve safety and guard against unforeseen consequences, a new build may be deployed in parallel to the current build in an A/B configuration, also called beta testing. This testing gradually expands to larger user groups until all users are on the new build. At that point, the previous build is retired, and its computing resources freed for other applications.

Tools: There are many dedicated deployment tools, including Ansible Tower, Chef, Codeship, ElectricFlow, Jenkins, Octopus Deploy, Spinnaker and TeamCity. Deployments also may involve application performance monitoring (APM) tools and other types of instrumentation to check and report on an application's health.

How do you implement a CI/CD pipeline?

There is no one way to set up a CI/CD pipeline. Precise steps vary between tools and the process to implement -- and that's by design, to tailor an incredibly agile pipeline that meets the needs of the business and its projects.

Still, there are common steps and decisions at each stage of pipeline construction that generally apply to any CI/CD process:

1. Select a version control system to maintain code repositories. Determine if you need a hosted version, or a hosting provider. Major cloud providers also offer options here, such as Azure DevOps.
2. Create repositories to house application source code and pipelines.
3. Determine what build, or CI, server to use. This can be self-hosted, such as Jenkins, or a third-party option such as GitHub Actions, CircleCI or Azure Pipelines.
4. Implement a task in the pipeline that compiles application source code into a build. In some setups, this will generate a Docker image.
5. Run basic tests on the code (static analysis, style checks) to ensure its quality and consistency with organizational guidelines.
6. The build should now generate an artifact, or container image, published to a store or registry.
7. Initiate further testing on the build, as listed previously (functional, security, user acceptance, etc.). If predetermined thresholds are not met, fail the stage. Publish results of tests and code coverage so they are easily available.
8. Once the software build passes tests, it is ready for final preparations to production deployment. This may include multiple staged environments, such as blue/green and canary deployments.

Note that CI/CD based in the cloud function basically the same but rely heavily on tools and services native to the cloud provider's platform. This may require specific steps. For example, here is a walkthrough to build a CI/CD pipeline based on Azure DevOps and GitHub.

Each stage of a typical CI/CD pipeline has multiple tasks and multiple kinds of tools to accomplish them.

CI/CD implementation encounters some specific challenges, too. Here are two major ones to watch out for:

• Limited testing. Allocating and coordinating resources and intellectual investment to configure test environments and construct test cases is a common problem for CI/CD pipelines. Continuous development involves multiple code commits and parallel testing demands that frequently result in configuration conflicts and limited/forgotten test cases. This can lead to errors slipping through the test phase and degrading the pipeline's efficiency. There's no substitute for skilled and knowledgeable software testers and well-documented requirements and goals.
• Fixing bugs. Pipelines are designed to provide feedback loops back to developers who can fix bugs in a new build. Finding a bug is easy enough, but it can be difficult to identify the specific developer responsible to fix that section of code. This makes it harder to hold developers responsible for their work and can obfuscate the need for more training. Logging, team communication and copious documentation can help determine the location of the bug and identify the developers to be involved in its resolution.

CI/CD pipeline best practices

A business and its development teams can employ various methods to get the most from a CI/CD pipeline. These CI/CD best practices can help organizations derive even more value from them.

Start small. CI/CD brings speed and agility, so give the process time to evolve and enable developers to try different tools and steps. A business might start with a CI pipeline and add CD later. Small filler projects are ideal places to try new tools and techniques that can enhance a broader pipeline.

Define success. Understand the intended benefits, such as faster code building or lower error/rework rates, and then implement metrics to measure those criteria. Compare the metrics against pre-pipeline performance and track those metrics as the pipeline evolves. This makes it easier to see the pipeline's value, spot problems over time, and invest in ways to build and enhance the CI/CD pipeline.

Document processes. Often overlooked and underappreciated, documentation is an essential part of the development pipeline. It lays out the process and tools for all developers and business users, and explains how everything is related and configured. Plus, it can help troubleshoot problems. Documentation also contributes to an organization's compliance and security posture, enabling leaders to audit activities.

Think about operations. Agile development paradigms, such as DevOps and continuous deployment, embrace both operations and development roles. Developers must understand both deployment and operations, and take greater ownership of the software's reliability and performance. Business and project leaders must foster and reinforce this attitude shift.

Focus on feedback. Feedback within the CI/CD pipeline is most effective when every step -- and every participant -- actively works to spot and address issues to save time and work efficiently. This starts with spotting errors in the source code and continues all the way through testing and deployment. For example, find and fix a syntax error in the source code at the build stage, rather than waste time and effort during the testing phase. Categorizing and analyzing errors can also help businesses improve the development skills and processes.

Infuse security throughout. Security scanning tools at the code level (SAST and SCA tools) are handy for early vulnerability and error diagnostics but can produce a large number of false positives. Security scanning at the test level (DAST and IAST tools) requires the software to be built and running, which means errors are caught later in the pipeline where bug fixes are more time-consuming and costly. Select the best security scanning tools for the tasks at hand, and use those tools to automatically update the bug tracking system and automatically generate tickets for fast examination and remediation.

Also, think about the security of the pipeline itself: Improper authorization and authentication can expose tools and repositories to malicious actions. Secure tools and repositories, use logs to track user access to them and flag unusual activities -- such as downloads to IP addresses outside of the corporate LAN.

Embrace continuous testing. During the source and build stages, perform SCA, SAST and other basic code scans for style and security standards. When the build is completed, apply a battery of established test conditions using test tools. Start with simple functional validation, and systematically expand testing to more complex and comprehensive integration, in-depth security (such as DAST) and performance. Staff must carefully construct tests and test cases to validate the features and functionality of each new build as the builds, the project and even the project's requirements evolve. Frequently added new features require frequent tests and test cases.

Foster communication. Integration and delivery work together but are often implemented separately and handled by different teams, such as coders and testers. A CI/CD pipeline that functions smoothly requires timely and clear communication and collaboration between different teams across the pipeline; otherwise, it can easily break down with unnecessary delays.

Avoid waste. Creating and maintaining a CI/CD pipeline incurs various costs for tools, infrastructure and resources. Inefficient use of any of these -- underutilized or unused tools, overallocated IT infrastructure for testing and deployments, poor coordination, insufficient evaluation of processes or tools -- can slow development and impair developer productivity. CI/CD pipelines are dynamic entities that require frequent refinement and regular developer training to operate efficiently and reliably.

This also means avoiding less obvious areas of inefficiencies. Don't make 10 different builds in the same day if there is no practical way to test and deploy those 10 builds in the same day. Teams and project effort must reflect the most effective use of the pipeline.

Enhance the ecosystem. CI/CD and other agile pipelines are ecosystems composed of tools tied together with processes and automation, with myriad alternate paths and steps for different products. Teams should always evaluate new tools and refine the processes to keep the overall pipeline as smooth and efficient as possible.

loyaharriew.blogspot.com

Source: https://www.techtarget.com/searchsoftwarequality/CI-CD-pipelines-explained-Everything-you-need-to-know