Merge pull request #4883 from mitchellh/gc-vagrant-1.7-ansible-provisioner

Non-trivial changes and bug fixes for the Ansible provisioner

CHANGELOG.md

@@ -1,5 +1,11 @@
 ## 1.7.0 (unreleased)
 
+BREAKING CHANGES:
+
+  - provisioners/ansible: `raw_arguments` has now highest priority
+  - provisioners/ansible: only the `ssh` connection transport is supported
+      (`paramiko` can be enabled with `raw_arguments` at your own risks)
+
 FEATURES:
 
   - **Named provisioners**: Provisioners can now be named. This name is used
@@ -88,6 +94,9 @@ BUG FIXES:
       IP address and don't allow it. [GH-4671]
   - providers/virtualbox: Show more descriptive error if VirtualBox is
       reporting an empty version. [GH-4657]
+  - provisioners/ansible: Force `ssh` (OpenSSH) connection by default [GH-3396]
+  - provisioners/ansible: Don't use or modify `~/.ssh/known_hosts` file by default,
+      similarly to native vagrant commands [GH-3900]
   - provisioners/docker: Get GPG key over SSL. [GH-4597]
   - provisioners/docker: Search for docker binary in multiple places. [GH-4580]
   - provisioners/salt: Highstate works properly with a master. [GH-4471]

plugins/provisioners/ansible/provisioner.rb

@@ -12,31 +12,24 @@ module VagrantPlugins
         @ssh_info = @machine.ssh_info
 
         #
-        # 1) Default Settings (lowest precedence)
+        # Ansible provisioner options
         #
 
         # Connect with Vagrant SSH identity
         options = %W[--private-key=#{@ssh_info[:private_key_path][0]} --user=#{@ssh_info[:username]}]
 
-        # Multiple SSH keys and/or SSH forwarding can be passed via
+        # Connect with native OpenSSH client
-        # ANSIBLE_SSH_ARGS environment variable, which requires 'ssh' mode.
+        # Other modes (e.g. paramiko) are not officially supported,
-        # Note that multiple keys and ssh-forwarding settings are not supported
+        # but can be enabled via raw_arguments option.
-        # by deprecated 'paramiko' mode.
+        options << "--connection=ssh"
-        options << "--connection=ssh" unless ansible_ssh_args.empty?
 
-        # By default we limit by the current machine.
+        # By default we limit by the current machine, but
-        # This can be overridden by the limit config option.
+        # this can be overridden by the `limit` option.
-        options << "--limit=#{@machine.name}" unless config.limit
+        if config.limit
-
+          options << "--limit=#{as_list_argument(config.limit)}"
-        #
+        else
-        # 2) Configuration Joker
+          options << "--limit=#{@machine.name}"
-        #
+        end
-
-        options.concat(self.as_array(config.raw_arguments)) if config.raw_arguments
-
-        #
-        # 3) Append Provisioner options (highest precedence):
-        #
 
         options << "--inventory-file=#{self.setup_inventory_file}"
         options << "--extra-vars=#{self.get_extra_vars_argument}" if config.extra_vars
@@ -48,22 +41,29 @@ module VagrantPlugins
         options << "--vault-password-file=#{config.vault_password_file}" if config.vault_password_file
         options << "--tags=#{as_list_argument(config.tags)}" if config.tags
         options << "--skip-tags=#{as_list_argument(config.skip_tags)}" if config.skip_tags
-        options << "--limit=#{as_list_argument(config.limit)}" if config.limit
         options << "--start-at-task=#{config.start_at_task}" if config.start_at_task
 
+        # Finally, add the raw configuration options, which has the highest precedence
+        # and can therefore potentially override any other options of this provisioner.
+        options.concat(self.as_array(config.raw_arguments)) if config.raw_arguments
+
+        #
         # Assemble the full ansible-playbook command
+        #
+
         command = (%w(ansible-playbook) << options << config.playbook).flatten
 
-        # Some Ansible options must be passed as environment variables
         env = {
-          "ANSIBLE_FORCE_COLOR" => "true",
-          "ANSIBLE_HOST_KEY_CHECKING" => "#{config.host_key_checking}",
-
           # Ensure Ansible output isn't buffered so that we receive output
           # on a task-by-task basis.
-          "PYTHONUNBUFFERED" => 1
+          "PYTHONUNBUFFERED" => 1,
+
+          # Some Ansible options must be passed as environment variables,
+          # as there is no equivalent command line arguments
+          "ANSIBLE_FORCE_COLOR" => "true",
+          "ANSIBLE_HOST_KEY_CHECKING" => "#{config.host_key_checking}",
         }
-        # Support Multiple SSH keys and SSH forwarding:
+        # ANSIBLE_SSH_ARGS is required for Multiple SSH keys, SSH forwarding and custom SSH settings
         env["ANSIBLE_SSH_ARGS"] = ansible_ssh_args unless ansible_ssh_args.empty?
 
         show_ansible_playbook_command(env, command) if (config.verbose || @logger.debug?)
@@ -183,9 +183,14 @@ module VagrantPlugins
         @ansible_ssh_args ||= get_ansible_ssh_args
       end
 
+      # Use ANSIBLE_SSH_ARGS to pass some OpenSSH options that are not wrapped by
+      # an ad-hoc Ansible option. Last update corresponds to Ansible 1.8
       def get_ansible_ssh_args
         ssh_options = []
 
+        # Don't access user's known_hosts file, except when host_key_checking is enabled.
+        ssh_options << "-o UserKnownHostsFile=/dev/null" unless config.host_key_checking
+
         # Multiple Private Keys
         @ssh_info[:private_key_path].drop(1).each do |key|
           ssh_options << "-o IdentityFile=#{key}"

test/unit/plugins/provisioners/ansible/provisioner_test.rb

@@ -62,13 +62,16 @@ VF
   # Class methods for code reuse across examples
   #
 
-  def self.it_should_set_arguments_and_environment_variables(expected_args_count = 5, expected_vars_count = 3, expected_host_key_checking = false)
+  def self.it_should_set_arguments_and_environment_variables(
+    expected_args_count = 6, expected_vars_count = 4, expected_host_key_checking = false, expected_transport_mode = "ssh")
+
     it "sets implicit arguments in a specific order" do
       expect(Vagrant::Util::Subprocess).to receive(:execute).with { |*args|
 
         expect(args[0]).to eq("ansible-playbook")
         expect(args[1]).to eq("--private-key=#{machine.ssh_info[:private_key_path][0]}")
         expect(args[2]).to eq("--user=#{machine.ssh_info[:username]}")
+        expect(args[3]).to eq("--connection=ssh")
 
         inventory_count = args.count { |x| x =~ /^--inventory-file=.+$/ }
         expect(inventory_count).to be > 0
@@ -79,18 +82,18 @@ VF
 
     it "sets --limit argument" do
       expect(Vagrant::Util::Subprocess).to receive(:execute).with { |*args|
-        raw_limits = []
+        all_limits = args.select { |x| x =~ /^(--limit=|-l)/ }
         if config.raw_arguments
           raw_limits = config.raw_arguments.select { |x| x =~ /^(--limit=|-l)/ }
-        end
+          expect(all_limits.length - raw_limits.length).to eq(1)
-        all_limits = args.select { |x| x =~ /^(--limit=|-l)/ }
+          expect(all_limits.last).to eq(raw_limits.last)
-        expect(all_limits.length - raw_limits.length).to eq(1)
-
-        if config.limit
-          limit = config.limit.kind_of?(Array) ? config.limit.join(',') : config.limit
-          expect(all_limits.last).to eq("--limit=#{limit}")
         else
-          expect(all_limits.first).to eq("--limit=#{machine.name}")
+          if config.limit
+            limit = config.limit.kind_of?(Array) ? config.limit.join(',') : config.limit
+            expect(all_limits.last).to eq("--limit=#{limit}")
+          else
+            expect(all_limits.first).to eq("--limit=#{machine.name}")
+          end
         end
       }
     end
@@ -98,6 +101,12 @@ VF
     it "exports environment variables" do
       expect(Vagrant::Util::Subprocess).to receive(:execute).with { |*args|
         cmd_opts = args.last
+
+        if expected_host_key_checking
+          expect(cmd_opts[:env]['ANSIBLE_SSH_ARGS']).to be_nil unless config.raw_arguments
+        else
+          expect(cmd_opts[:env]['ANSIBLE_SSH_ARGS']).to include("-o UserKnownHostsFile=/dev/null")
+        end
         expect(cmd_opts[:env]['ANSIBLE_FORCE_COLOR']).to eql("true")
         expect(cmd_opts[:env]['ANSIBLE_HOST_KEY_CHECKING']).to eql(expected_host_key_checking.to_s)
         expect(cmd_opts[:env]['PYTHONUNBUFFERED']).to eql(1)
@@ -111,6 +120,15 @@ VF
         expect(args.last[:env].length).to eq(expected_vars_count)
       }
     end
+
+    it "enables '#{expected_transport_mode}' transport mode" do
+      expect(Vagrant::Util::Subprocess).to receive(:execute).with { |*args|
+        index = args.rindex("--connection=#{expected_transport_mode}")
+        expect(index).to be > 0
+        expect(find_last_argument_after(index, args, /--connection=\w+/)).to be_false
+      }
+    end
+
   end
 
   def self.it_should_set_optional_arguments(arg_map)
@@ -128,35 +146,7 @@ VF
     end
   end
 
-  def self.it_should_use_smart_transport_mode
+  def self.it_should_explicitly_enable_ansible_ssh_control_persist_defaults
-    it "does not export ANSIBLE_SSH_ARGS" do
-      expect(Vagrant::Util::Subprocess).to receive(:execute).with { |*args|
-        cmd_opts = args.last
-        expect(cmd_opts[:env]['ANSIBLE_SSH_ARGS']).to be_nil
-      }
-    end
-
-    it "does not force any transport mode" do
-      expect(Vagrant::Util::Subprocess).to receive(:execute).with { |*args|
-        total = args.count { |x| x =~ /^--connection=\w+$/ }
-        expect(total).to eql(0)
-      }
-    end
-  end
-
-  def self.it_should_use_transport_mode(transport_mode)
-    it "enables '#{transport_mode}' transport mode" do
-      expect(Vagrant::Util::Subprocess).to receive(:execute).with { |*args|
-        index = args.rindex("--connection=#{transport_mode}")
-        expect(index).to be > 0
-        expect(find_last_argument_after(index, args, /--connection=\w+/)).to be_false
-      }
-    end
-  end
-
-  def self.it_should_force_ssh_transport_mode
-    it_should_use_transport_mode('ssh')
-
     it "configures ControlPersist (like Ansible defaults) via ANSIBLE_SSH_ARGS" do
       expect(Vagrant::Util::Subprocess).to receive(:execute).with { |*args|
         cmd_opts = args.last
@@ -212,7 +202,6 @@ VF
 
     describe "with default options" do
       it_should_set_arguments_and_environment_variables
-      it_should_use_smart_transport_mode
       it_should_create_and_use_generated_inventory
 
       it "does not add any group section to the generated inventory" do
@@ -281,8 +270,7 @@ VF
         config.host_key_checking = true
       end
 
-      it_should_set_arguments_and_environment_variables 5, 3, true
+      it_should_set_arguments_and_environment_variables 6, 3, true
-      it_should_use_smart_transport_mode
     end
 
     describe "with boolean (flag) options disabled" do
@@ -294,7 +282,7 @@ VF
         config.sudo_user = 'root'
       end
 
-      it_should_set_arguments_and_environment_variables 6
+      it_should_set_arguments_and_environment_variables 7
       it_should_set_optional_arguments({ "sudo_user" => "--sudo-user=root" })
 
       it "it does not set boolean flag when corresponding option is set to false" do
@@ -310,6 +298,7 @@ VF
       before do
         config.sudo = false
         config.skip_tags = %w(foo bar)
+        config.limit = "all"
         config.raw_arguments = ["--connection=paramiko",
                                 "--skip-tags=ignored",
                                 "--module-path=/other/modules",
@@ -318,12 +307,11 @@ VF
                                 "--limit=foo",
                                 "--limit=bar",
                                 "--inventory-file=/forget/it/my/friend",
+                                "--user=lion",
                                 "--new-arg=yeah"]
       end
 
-      it_should_set_arguments_and_environment_variables 15
+      it_should_set_arguments_and_environment_variables 17, 4, false, "paramiko"
-      it_should_create_and_use_generated_inventory
-      it_should_use_transport_mode('paramiko')
 
       it "sets all raw arguments" do
         expect(Vagrant::Util::Subprocess).to receive(:execute).with { |*args|
@@ -333,9 +321,12 @@ VF
         }
       end
 
-      it "sets raw arguments before arguments related to supported options" do
+      it "sets raw arguments after arguments related to supported options" do
         expect(Vagrant::Util::Subprocess).to receive(:execute).with { |*args|
-          expect(args.index("--skip-tags=foo,bar")).to be > args.index("--skip-tags=ignored")
+          expect(args.index("--user=lion")).to be > args.index("--user=testuser")
+          expect(args.index("--inventory-file=/forget/it/my/friend")).to be > args.index("--inventory-file=#{generated_inventory_dir}")
+          expect(args.index("--limit=bar")).to be > args.index("--limit=all")
+          expect(args.index("--skip-tags=ignored")).to be > args.index("--skip-tags=foo,bar")
         }
       end
 
@@ -361,7 +352,6 @@ VF
       end
 
       it_should_set_arguments_and_environment_variables
-      it_should_use_smart_transport_mode
 
       it "does not generate the inventory and uses given inventory path instead" do
         expect(Vagrant::Util::Subprocess).to receive(:execute).with { |*args|
@@ -377,7 +367,7 @@ VF
         config.ask_vault_pass = true
       end
 
-      it_should_set_arguments_and_environment_variables 6
+      it_should_set_arguments_and_environment_variables 7
 
       it "should ask the vault password" do
         expect(Vagrant::Util::Subprocess).to receive(:execute).with { |*args|
@@ -391,7 +381,7 @@ VF
         config.vault_password_file = existing_file
       end
 
-      it_should_set_arguments_and_environment_variables 6
+      it_should_set_arguments_and_environment_variables 7
 
       it "uses the given vault password file" do
         expect(Vagrant::Util::Subprocess).to receive(:execute).with { |*args|
@@ -406,7 +396,7 @@ VF
       end
 
       it_should_set_arguments_and_environment_variables 6, 4
-      it_should_force_ssh_transport_mode
+      it_should_explicitly_enable_ansible_ssh_control_persist_defaults
 
       it "passes custom SSH options via ANSIBLE_SSH_ARGS with the highest priority" do
         expect(Vagrant::Util::Subprocess).to receive(:execute).with { |*args|
@@ -440,7 +430,7 @@ VF
       end
 
       it_should_set_arguments_and_environment_variables 6, 4
-      it_should_force_ssh_transport_mode
+      it_should_explicitly_enable_ansible_ssh_control_persist_defaults
 
       it "passes additional Identity Files via ANSIBLE_SSH_ARGS" do
         expect(Vagrant::Util::Subprocess).to receive(:execute).with { |*args|
@@ -457,7 +447,7 @@ VF
       end
 
       it_should_set_arguments_and_environment_variables 6, 4
-      it_should_force_ssh_transport_mode
+      it_should_explicitly_enable_ansible_ssh_control_persist_defaults
 
       it "enables SSH-Forwarding via ANSIBLE_SSH_ARGS" do
         expect(Vagrant::Util::Subprocess).to receive(:execute).with { |*args|
@@ -475,7 +465,7 @@ VF
 
       it "shows the ansible-playbook command" do
         expect(machine.env.ui).to receive(:detail).with { |full_command|
-          expect(full_command).to eq("ANSIBLE_FORCE_COLOR=true ANSIBLE_HOST_KEY_CHECKING=false PYTHONUNBUFFERED=1 ansible-playbook --private-key=/path/to/my/key --user=testuser --limit='machine1' --inventory-file=#{generated_inventory_dir} playbook.yml")
+          expect(full_command).to eq("PYTHONUNBUFFERED=1 ANSIBLE_FORCE_COLOR=true ANSIBLE_HOST_KEY_CHECKING=false ANSIBLE_SSH_ARGS='-o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o ControlPersist=60s' ansible-playbook --private-key=/path/to/my/key --user=testuser --connection=ssh --limit='machine1' --inventory-file=#{generated_inventory_dir} playbook.yml")
         }
       end
     end
@@ -485,12 +475,12 @@ VF
         config.verbose = 'v'
       end
 
-      it_should_set_arguments_and_environment_variables 6
+      it_should_set_arguments_and_environment_variables 7
       it_should_set_optional_arguments({ "verbose" => "-v" })
 
       it "shows the ansible-playbook command" do
         expect(machine.env.ui).to receive(:detail).with { |full_command|
-          expect(full_command).to eq("ANSIBLE_FORCE_COLOR=true ANSIBLE_HOST_KEY_CHECKING=false PYTHONUNBUFFERED=1 ansible-playbook --private-key=/path/to/my/key --user=testuser --limit='machine1' --inventory-file=#{generated_inventory_dir} -v playbook.yml")
+          expect(full_command).to eq("PYTHONUNBUFFERED=1 ANSIBLE_FORCE_COLOR=true ANSIBLE_HOST_KEY_CHECKING=false ANSIBLE_SSH_ARGS='-o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o ControlPersist=60s' ansible-playbook --private-key=/path/to/my/key --user=testuser --connection=ssh --limit='machine1' --inventory-file=#{generated_inventory_dir} -v playbook.yml")
         }
       end
     end
@@ -524,7 +514,7 @@ VF
       end
 
       it_should_set_arguments_and_environment_variables 20, 4, true
-      it_should_force_ssh_transport_mode
+      it_should_explicitly_enable_ansible_ssh_control_persist_defaults
       it_should_set_optional_arguments({  "extra_vars"          => "--extra-vars=@#{File.expand_path(__FILE__)}",
                                           "sudo"                => "--sudo",
                                           "sudo_user"           => "--sudo-user=deployer",
@@ -537,7 +527,7 @@ VF
                                           "limit"               => "--limit=machine*:&vagrant:!that_one",
                                           "start_at_task"       => "--start-at-task=an awesome task",
                                         })
-    
+
       it "also includes given raw arguments" do
         expect(Vagrant::Util::Subprocess).to receive(:execute).with { |*args|
           expect(args).to include("--su-user=foot")
@@ -548,7 +538,7 @@ VF
 
       it "shows the ansible-playbook command, with additional quotes when required" do
         expect(machine.env.ui).to receive(:detail).with { |full_command|
-          expect(full_command).to eq("ANSIBLE_FORCE_COLOR=true ANSIBLE_HOST_KEY_CHECKING=true PYTHONUNBUFFERED=1 ANSIBLE_SSH_ARGS='-o IdentityFile=/my/key2 -o ForwardAgent=yes -o ControlMaster=no -o ControlMaster=auto -o ControlPersist=60s' ansible-playbook --private-key=/my/key1 --user=testuser --connection=ssh --why-not --su-user=foot --ask-su-pass --limit='all' --inventory-file=#{generated_inventory_dir} --extra-vars=@#{File.expand_path(__FILE__)} --sudo --sudo-user=deployer -vvv --ask-sudo-pass --ask-vault-pass --vault-password-file=#{File.expand_path(__FILE__)} --tags=db,www --skip-tags=foo,bar --limit='machine*:&vagrant:!that_one' --start-at-task='an awesome task' playbook.yml")
+          expect(full_command).to eq("PYTHONUNBUFFERED=1 ANSIBLE_FORCE_COLOR=true ANSIBLE_HOST_KEY_CHECKING=true ANSIBLE_SSH_ARGS='-o IdentityFile=/my/key2 -o ForwardAgent=yes -o ControlMaster=no -o ControlMaster=auto -o ControlPersist=60s' ansible-playbook --private-key=/my/key1 --user=testuser --connection=ssh --limit='machine*:&vagrant:!that_one' --inventory-file=#{generated_inventory_dir} --extra-vars=@#{File.expand_path(__FILE__)} --sudo --sudo-user=deployer -vvv --ask-sudo-pass --ask-vault-pass --vault-password-file=#{File.expand_path(__FILE__)} --tags=db,www --skip-tags=foo,bar --start-at-task='an awesome task' --why-not --su-user=foot --ask-su-pass --limit='all' playbook.yml")
         }
       end
     end

website/docs/source/v2/provisioning/ansible.html.md

@@ -8,7 +8,7 @@ sidebar_current: "provisioning-ansible"
 **Provisioner name: `"ansible"`**
 
 The ansible provisioner allows you to provision the guest using
-[Ansible](http://ansible.com) playbooks.
+[Ansible](http://ansible.com) playbooks by executing `ansible-playbook` from the Vagrant host.
 
 Ansible playbooks are [YAML](http://en.wikipedia.org/wiki/YAML) documents that
 comprise the set of steps to be orchestrated on one or more machines. This documentation
@@ -25,6 +25,11 @@ a single page of documentation.
   </p>
 </div>
 
+## Setup Requirements
+
+* [Install Ansible](http://docs.ansible.com/intro_installation.html#installing-the-control-machine) on your Vagrant host.
+* Your Vagrant host should ideally provide a recent version of OpenSSH that [supports ControlPersist](http://docs.ansible.com/faq.html#how-do-i-get-ansible-to-reuse-connections-enable-kerberized-ssh-or-have-ansible-pay-attention-to-my-local-ssh-config-file)
+
 ## Inventory File
 
 When using Ansible, it needs to know on which machines a given playbook should run. It does
@@ -191,11 +196,9 @@ by the sudo command.
 * `ansible.tags` can be set to a string or an array of tags. Only plays, roles and tasks tagged with these values will be executed.
 * `ansible.skip_tags` can be set to a string or an array of tags. Only plays, roles and tasks that *do not match* these values will be executed.
 * `ansible.start_at_task` can be set to a string corresponding to the task name where the playbook provision will start.
-* `ansible.raw_arguments` can be set to an array of strings corresponding to a list of `ansible-playbook` arguments (e.g. `['--check', '-M /my/modules']`). It is an *unsafe wildcard* that can be used to apply Ansible options that are not (yet) supported by this Vagrant provisioner. Following precedence rules apply:
+* `ansible.raw_arguments` can be set to an array of strings corresponding to a list of `ansible-playbook` arguments (e.g. `['--check', '-M /my/modules']`). It is an *unsafe wildcard* that can be used to apply Ansible options that are not (yet) supported by this Vagrant provisioner. As of Vagrant 1.7, `raw_arguments` has the highest priority and its values can potentially override or break other Vagrant settings.
-  * Any supported options (described above) will override conflicting `raw_arguments` value (e.g. `--tags` or `--start-at-task`)
-  * Vagrant default user authentication can be overridden via `raw_arguments` (with custom values for `--user` and `--private-key`)
 * `ansible.raw_ssh_args` can be set to an array of strings corresponding to a list of OpenSSH client parameters (e.g. `['-o ControlMaster=no']`). It is an *unsafe wildcard* that can be used to pass additional SSH settings to Ansible via `ANSIBLE_SSH_ARGS` environment variable.
-* `ansible.host_key_checking` can be set to `true` which will enable host key checking. As Vagrant 1.5, the default value is `false`, to avoid connection problems when creating new virtual machines.
+* `ansible.host_key_checking` can be set to `true` which will enable host key checking. As of Vagrant 1.5, the default value is `false` and as of Vagrant 1.7 the user kownn host file (e.g. `~/.ssh/known_hosts`) is no longer read nor modified. In other words: by default, the Ansible provisioner behaves the same as Vagrant native commands (e.g `vagrant ssh`).
 
 ## Tips and Tricks