Tag

continuous integration

Agile, Continuous Delivery

Why do DevOps?

Background:

According to the Puppet’s 2016 State of DevOps Report?DevOps?is no longer a mere fad or buzz word but an understood set of practices and cultural patterns.

DevOps was originally coined by Patrick Debois and Andrew Shafer in 2008. The velocity conference??community made DevOps a known term with the famous 10+ deploys per day: Dev and Ops cooperation at Flickr?in 2009.

What is DevOps?:

Daniel Greene?in techcrunch?defines DevOps as a set of practices, tool and policies that lead to improved quality and automated delivery. But I really like Gene Kim’s definition in his book the phoenix project?where he refers to DevOps as the outcome of applying Lean principles to the IT value stream. These principles are age old but are now used to accelerate flow of work?in software development?through product management, development, test, operations and Information Security. ?Adapting lean principles and shift left, Toyota’s mantra is to stop and fix quality issues before they manifest at the end and also deliver just in time parts and support.

ProductionSystem_tcm-11-406919

Source:?Toyota website

Imagine applying these principles to a software development lifecycle and while coding and running automated testing, checking for known vulnerabilities, testing for security, continuously integrating code and deploying several times a day in various environments including production. This is shifting left the activities that were performed at the very end, which caused problems to be discovered late and teams could never build in quality. This lead to developers spending less time reworking their code while Ops spending time in late nights and weekends to deploy code into production. In fact according to the phoenix project, DevOps has benefited from an astounding convergence of philosophical movements such as Lean Startup, Innovation Culture, Toyota Kata, Rugged Computing and the Velocity community.

What do the development and operations typically see?:

What Operations sees? What development sees?
Fragile applications are prone to failure More urgent, date-driven projects put into the queue
Long time required to figure out ?which bit got flipped? Even more fragile code put into production
Detective control is a salesperson More releases have increasingly ?turbulent installs?
Too much time required to restore service Release cycles lengthen to amortize ?cost of deployments?
Too much firefighting and unplanned work Failing bigger deployments more difficult to diagnose
Planned project work cannot complete ?Most senior and constrained IT ops resources have less time to fix underlying process problems
?Frustrated customers leave ?Ever increasing backlog of infrastructure projects that could fix root cause and reduce costs
?Market share goes down ?Ever increasing amount of tension between IT Ops and Development

 

Source:?2011 06 15 velocity conf from visible ops to dev ops final

The above leads to a mindset that eventually breeds mistrust and a not so healthy competition between these teams.

Lets see how DevOps changes this:

Looking at some statistics of companies showed high performers delivered on faster time to market, increased customer satisfaction and market share, team and employee productivity and happiness as well as allowing organizations to win in the marketplace.

Amazon in 2011 deployed code every 11.6 seconds, Netflix several times a day and Google several time a week. ?Gary Gruver’s description of their transformation of HP’s laserjet division where?400 people distributed across 300 continents is able to integrate around 150 changes or 100,000 lines of code ?into the trunk on their 10m lines of code base every day.

BenefitsofDevOps

We know our quality within 24 hours of any fix going into the system and we can broadly test for even last minute fixes to ensure that a bug fix does not cause unexpected failures

Check out Guy Gruver’s talk?in 2013?on how they did it.

Summary:

DevOps transformation at scale is hard and needs changes in both organization and technology bringing together teams that have been in the past operated separately and introduces innovation?that automates end to end software development pipeline.

Continuous Delivery

Catch your continuous integration and deployment pipeline today

My review of Bitbucket pipeline beta

Screen Shot 2016-07-24 at 4.30.21 PM

So you would like to build, test and deploy your code continuously? but don’t want to install and configure your local instances of Jenkins or Bamboo (Check out our thoughts on both tools), you don’t have to anymore. Atlassian is replacing its Bamboo on Cloud offering with Bitbucket pipeline. This offering is for small and medium sized businesses.

I recently signed up for the Bitbucket pipeline beta program at bitbucket.org and got accepted a month ago. Read on for my sum up of this new Atlassian offering

Key features:

  1. A continuous integration (CI) and Continuous deployment (CD) engine is ready for use on any of your git or mercurial repositories by just adding a bitbucket-pipeline.yml file in your project root. You pipeline is triggered as soon a change is pushed to any or a selected branch. See the below simple example of the bitbucket-pipeline.yml file.
    image: node:5.11.0
    pipelines:
    ? default:
    ??? - step:
    ??????? script:
    ????????? - echo "This script runs on all branches that don't have any specific pipeline assigned in 'branches'."
    ? branches:
    ??? master:
    ????? - step:
    ????????? script:
    ??????????? - echo "This script runs only on commit to the master branch."
    ??? feature/*:
    ????? - step:
    ????????? image: java:openjdk-9 # This step uses its own image
    ????????? script:
    ??????????? - echo "This script runs only on commit to branches with names that match the feature/* pattern."
  2. Employs docker images to build, test and deploy your code. Out of the box has images for the following languages Screen Shot 2016-07-24 at 6.07.10 PM?
  3. Support for builds in any docker image. You can build your own docker image with your language support and build it using the pipeline. Docker images for additional languages below are available at? docker hub.

    I needed a docker image that has the atlassian sdk that is built on top of maven to build my sample atlassian plugin. I used a docker image built by Translucent. You can get it at the?docker hub. Take a look at my bitbucket.pipeline.yml file below or get the entire source at github

    image: translucent/atlassian-plugin-sdk
    
    pipelines:
      default:
        - step:
            script: # Modify the commands below to build your repository.
              - cd adminUI
              - pwd
              - atlas-version
              - atlas-mvn clean install
  4. Environment variables allow passing security parameters and other configurable variables.? You can write scripts in the yml file or invoke nested scripts from your project folder.
  5. Community support is increasing and well documented
  6. Several integration images are available to test, monitor and deploy your code.Screen Shot 2016-07-24 at 6.26.01 PM

The appeal:

  • Simple to configure and you can start building with a flip of a switch
  • Simple architecture and ability to invoke parallel scripts and trigger different docker images delivers great scalability
  • Industry supported docker images and ability to create your own images makes this offering flexible

?Limitations:

  • No Email and HipChat notifications
  • No Jira integration
  • Build speeds and concurrency is limited

The product is still in Beta, but will appeal to most due to its simplicity and integrated seamlessly into the Bitbucket UI.

Feel free to share your experience

 

Continuous Delivery

Look at Bamboo with Jenkins fan’s eyes

For the past 10 years, my projects’?Continuous Integration were done on Hudson/Jenkins. Cannot say I’d ever been a Jenkins fan though, but I haven’t had a chance to work?with?alternatives either. Until I stumbled over Atlassian Bamboo.
At the end of the day, both?Bamboo and Jenkins are Continuous Integration (CI) tools,?but while Jenkins is doing its best to support Continuous Delivery (CD) via newly developed plugins, Bamboo is built with CD in?its backbone.?

?Native support for Continuous Delivery?Pipeline

Bamboo CD pipeline is a chain of stages. Each stage contains 1 ore more jobs. Each job consists of one or more tasks.
For example, a project continuous delivery pipeline consist of the following stages: package, deploy to QA, run tests in QA, deploy to UAT, smoke tests in UAT.
“Package” stage contains one “Build and Package” job that has “compile”, “unit test”, and “install into Artifactory” tasks.
“Run Tests in QA” stage contains 2 jobs that are executed in parallel: “Run Regression tests” and “Run Functional tests?.
All jobs in one stage have to be completed successfully before moving to the next stage.
Steps can be configured to pause for manual input.
As a part of a pipeline, a release candidate can be manually approved and promoted to higher environments. For example, a build that successfully passed all stages including a smoke test in UAT can be marked as approved by UAT team and promoted to Prod by Operate team. Bamboo groups permissions would ensure correct access to these functions.
?


Many daily CI features are put on auto-pilot:

  1. Choose one of two merge models to?avoid manual merging?? to trigger a feature build, Bamboo will use either the ?Branch Updater? to update a feature branch with the integration branch or the ?Gatekeeper? to update the integration with the feature branch.
  2. Enable ?push on success? to merge a pull request to the feature branch via Branch Updater or to the integration branch via the Gatekeeper.
  3. Configure Bamboo to automatically delete configurations for branches that are deleted in Stash

Tight integration with JIRA:

  1. Generate Release Notes for each build and shows a difference between current and any previous builds.
  2. Link issues to dev branches to see real-time build status inside the issue
  3. Get related build results inside issues and the issue’s vital stats on the build result in Bamboo.
  4. Track each issue’s deployment status throughout your environments.
  5. See a roll-up of all the JIRA?issues and commits included in each release candidate.
  6. Create an issue from Bamboo failed job to resolve it
?


Tight support of Docker build agents and tasks:

  1. Build a Docker image, run a Docker container, and push the image to Docker Registry
  2. Run as a Bamboo Agent inside a Docker container to ease up distribution of changes to Bamboo instances and avoid conflicts between remote agents on the same host
?