ganeti-github.git
18 months agoAdd SkipTest in hv_kvm psutil-specific test cases master
Federico 'Morg' Pareschi [Sun, 4 Feb 2018 13:10:59 +0000 (08:10 -0500)]
Add SkipTest in hv_kvm psutil-specific test cases

Signed-off-by: Federico Morg Pareschi <morg@google.com>

18 months agoA few wording fixes to the 'ifdown' design (#1280)
Iustin Pop [Sun, 4 Feb 2018 07:50:01 +0000 (08:50 +0100)]
A few wording fixes to the 'ifdown' design (#1280)

These are not changing the design, just making the document easier to
read. Also re-wrap design-ifdown document at 72 chars per line.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Federico Morg Pareschi <morg@google.com>

18 months agoTypos in gnt-instance man page (#1279)
Phil Regnauld [Sat, 3 Feb 2018 10:47:37 +0000 (11:47 +0100)]
Typos in gnt-instance man page (#1279)

23 months agoFix #1260 ganeti-cleaner accepts REMOVE_AFTER as env var (#1267)
Calum Calder [Tue, 19 Sep 2017 11:23:26 +0000 (12:23 +0100)]
Fix #1260 ganeti-cleaner accepts REMOVE_AFTER as env var (#1267)

This commit adds the option to use REMOVE_AFTER env var before running ganeti-cleaner master
to specify how many days to use as threshold before removal.

Signed-off-by: Calum Calder <calumcalder@google.com>
Reviewed-by: Federico Morg Pareschi <morg@google.com>

2 years agoFix --verify-ssh-clutter - fixes #1255 (#1264)
Calum Calder [Mon, 21 Aug 2017 13:28:32 +0000 (14:28 +0100)]
Fix --verify-ssh-clutter - fixes #1255 (#1264)

Fix for gnt-cluster verify --verify-ssh-clutter incorrectly parsing data
received over RPC.

2 years agoMove postcopy migration logic to cmdlib (#1263)
Calum Calder [Wed, 16 Aug 2017 12:57:28 +0000 (13:57 +0100)]
Move postcopy migration logic to cmdlib (#1263)

Exposed hypervisor postcopy functions via RPC, and implemented logic to execute them in the MigrateInstance tasklet.

Signed-off-by: Calum Calder <calumcalder@google.com>
Reviewed-by: Federico Morg Pareschi <morg@google.com>

2 years agonetutils: use correct syntax of iproute2 when getting ip address (#26)
Arnd Hannemann [Wed, 9 Aug 2017 09:44:19 +0000 (11:44 +0200)]
netutils: use correct syntax of iproute2 when getting ip address (#26)

Before this patch there was a bug when the interface name was actually "dev":

2016-10-11 19:45:05,025: ganeti-noded pid=15175 INFO RunCmd /sbin/ip -o addr show dev
2016-10-11 19:45:05,027: ganeti-noded pid=15175 ERROR Error running the ip command while getting the IP addresses of dev
2016-10-11 19:45:05,027: ganeti-noded pid=15175 ERROR Error in RPC call
Traceback (most recent call last):
  File "/usr/share/ganeti/2.12/ganeti/server/noded.py", line 191, in HandleRequest
    result = (True, method(serializer.LoadJson(req.request_body)))
  File "/usr/share/ganeti/2.12/ganeti/server/noded.py", line 627, in perspective_instance_start
    return backend.StartInstance(instance, startup_paused, trail)
  File "/usr/share/ganeti/2.12/ganeti/backend.py", line 1892, in StartInstance
    hyper.StartInstance(instance, block_devices, startup_paused)
  File "/usr/share/ganeti/2.12/ganeti/hypervisor/hv_kvm/__init__.py", line 1775, in StartInstance
    startup_paused, kvmhelp)
  File "/usr/share/ganeti/2.12/ganeti/hypervisor/hv_kvm/__init__.py", line 1269, in _GenerateKVMRuntime
    elif (addresses[constants.IP4_VERSION] and
TypeError: 'NoneType' object has no attribute '__getitem__'

The previous code only worked, because iproute2 is liberate in what it
excepts. However the correct syntax is:
ip address { show | save | flush } [ dev IFNAME ]

This patch fixes the bug by using the correct syntax.

Signed-off-by: Arnd Hannemann <arnd@arndnet.de>
Reviewed-by: Federico Morg Pareschi <morg@google.com>

2 years agoImplement basic automatic KVM postcopy migration (#1262)
Calum Calder [Tue, 1 Aug 2017 10:20:19 +0000 (11:20 +0100)]
Implement basic automatic KVM postcopy migration (#1262)

This commit adds postcopy functionality to ganeti by automatically
switching a migration to postcopy mode if the postcopy-ram or
x-postcopy-ram migration capability flag is set in an instance's
hypervisor parameters.

Signed-off-by: Calum Calder <calumcalder@google.com>
Reviewed-by: Federico Morg Pareschi <morg@google.com>

2 years agoDisable CPU pinning on migration with cpu_mask=all (#1261)
Calum Calder [Tue, 1 Aug 2017 08:03:10 +0000 (09:03 +0100)]
Disable CPU pinning on migration with cpu_mask=all (#1261)

Setting the cpu_mask hypervisor parameter with cpu_mask=all disables
CPU pinning, allowing Ganeti to run without dependence on psutil.

Reviewed-by: Federico Morg Pareschi <morg@google.com>

2 years agoAllow IPv6 for vnc_bind_address (#1257)
Brian Candler [Tue, 11 Jul 2017 16:17:42 +0000 (17:17 +0100)]
Allow IPv6 for vnc_bind_address (#1257)

e.g. gnt-cluster modify -H kvm:vnc_bind_address=::

Also fixes incorrect error message for invalid vnc_bind_address.

Note: this does not affect behaviour when you specify an interface
name, which still just binds to the first IPv4 address found.

Reviewed-by: Federico Morg Pareschi <morg@google.com>

2 years agokvm: Make use of the -D option (#45)
Yiannis Tsiouris [Fri, 2 Jun 2017 12:46:27 +0000 (15:46 +0300)]
kvm: Make use of the -D option (#45)

Use a dedicated directory for the QEMU logs (i.e.
/var/log/ganeti/kvm). Along with the -daemonize option
pass the -D option in the KVM command line. QEMU instead
of redirecting stdout/stderr to /dev/null will use the
specified file (i.e. /var/log/ganeti/kvm/<instance name>.log).

Signed-off-by: Dimitris Aragiorgis <dimitris.aragiorgis@gmail.com>
Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Reviewed-by: Federico Morg Pareschi <morg@google.com>

2 years agoForce cache=none for KVM when aio=native (#43)
Alexandros Kosiaris [Thu, 1 Jun 2017 09:53:05 +0000 (12:53 +0300)]
Force cache=none for KVM when aio=native (#43)

Starting with version 2.6, QEMU will fail to start with an error when
aio=native and no cache=none is present. This is documented in
http://wiki.qemu.org/ChangeLog/2.6. QEMU has been complaining about this
since 2.3. Detect this in the hypervisor setup and force the disk_cache
setting, logging it at the same time.

Signed-off-by: Alexandros Kosiaris <akosiaris@gmail.com>
Reviewed-by: Federico Morg Pareschi <morg@google.com>

2 years agoExpose allow-failover option to RAPI client
Yiannis Tsiouris [Mon, 6 Feb 2017 14:17:11 +0000 (16:17 +0200)]
Expose allow-failover option to RAPI client

'gnt-instance migrate' supports performing a failover on an instance
that is not possible to migrate (e.g. if the instance is down). This
patch exposes this option ('--allow-failover') to the RAPI client.

Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Signed-off-by: Alex Kiousis <alexk@noc.grnet.gr>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

2 years agoImplement predictive queue cluster parameter
Federico Morg Pareschi [Mon, 9 Jan 2017 11:17:58 +0000 (11:17 +0000)]
Implement predictive queue cluster parameter

This commit adds the enabled_predictive_queue cluster parameter that
allows the cluster to specify if the predictive scheduler should be used
or not when ordering ganeti jobs in the queue. It also adds the correct
command line flags for the cluster init and cluster modify ganeti jobs.

As a default, if no option is specified upon cluster creation, the
predictive scheduler is automatically enabled.

Signed-off-by: Federico Morg Pareschi <morg@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

2 years agoFix incorrect 'any' function call
Federico Morg Pareschi [Wed, 4 Jan 2017 15:11:41 +0000 (15:11 +0000)]
Fix incorrect 'any' function call

As introduced by commit 77807d83 and d388e30c6d
and was not caught during the code review process.

Signed-off-by: Federico Morg Pareschi <morg@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoFIX: Add missing LockDecls files to Makefile.am
Federico Morg Pareschi [Mon, 19 Dec 2016 16:03:15 +0000 (16:03 +0000)]
FIX: Add missing LockDecls files to Makefile.am

Trivial change to add the missing LockDecls.hs files to Makefile.am.
This should remove some complaints to some of the tests that failed to
build.

Signed-off-by: Federico Morg Pareschi <morg@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoQuell remaining pylint 1.6.4 warnings after merge
Brian Foley [Mon, 19 Dec 2016 14:19:09 +0000 (14:19 +0000)]
Quell remaining pylint 1.6.4 warnings after merge

These should all be harmless modifications with no functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoMerge branch 'stable-2.17'
Brian Foley [Mon, 19 Dec 2016 12:02:39 +0000 (12:02 +0000)]
Merge branch 'stable-2.17'

* stable-2.17
  1143: Add gnt-cluster modify --modify-ssh-setup option

* stable-2.15
  Fix gnt-instance console instance unpausing for xl toolstack
  Disable pylint too-many-nested-blocks in _RunCmdPipe
  Reduce nesting in import-export ProcessChildIO
  Reduce nesting in LUOobCommand.Exec
  Reduce nesting in LUInstanceCreate.RunOsScripts
  Reduce nesting in RemoveNodeSshKeyBulk key calculation
  Reduce nesting in RemoveNodeSshKeyBulk ssh logic
  Reduce nesting in gnt-cluster VerifyDisks missing disk loop
  Reduce nesting in _CheckVLANArguments
  Reduce nesting in StartDaemon
  Disable pylint bad-continuation warning
  Disable pylint superfluous-parens warning
  Disable pylint redefined-variable-type warning
  Disable pylint too-many-branches warnings
  Disable pylint broad-except warnings
  Disable incorrect pylint assigning-non-slot warning
  Quell pylint unbalanced-tuple-unpacking warning
  Cleanup: Use new-style classes everywhere
  Quell pylint socket.timeout warning
  Quell the pylint wrong-import-order warnings
  Quell cell-var-from-loop warning
  Use default value lambda param to avoid cell-var-from-loop
  Quell too-many-boolean-expressions
  Remove pylint tests removed in pylint 2.0
  Quell trailing newline
  Quell bad-whitespace warning
  Quell consider-using-enumerate warning
  Disable pylint unsubscriptable-object warning
  Disable pylint bare-except warning
  Disable unwanted pylint wrong-import-position warnings
  Disable pylint unused-wildcard-import warning
  Disable incorrect pylint not-callable warning
  Disable pylint unpacking-non-sequence warning
  Disable pylint misplaced-comparison-constant warning
  Disable incorect pylint simplify-if-statement warning
  Disable pylint eval-used warning
  Disable pylint invalid-name warning
  Disable pylint import-self warning
  Disable some pylint unused-import warnings
  Replace deprecated pylint >=0.27 pragma with new form
  Delete old warning disables removed from pylint 1.6
  Fix pylint >1.4 pycurl no-member warnings
  Cleanup: Remove unused/duplicate module/fn import
  Cleanup: Fix unidiomatic-typecheck
  Cleanup: Remove some unneeded pylint disables
  Cleanup: Iterate dict rather than key list
  Cleanup: Remove unused format key
  Cleanup: StartInstance and RebootInstance return None
  Cleanup: Fix for/else with no break in AddAuthorizedKeys
  Cleanup: Replace map/filters with list comprehensions
  Cleanup: del is a statement not a function
  Cleanup: Use FOO not in BAR instead of not FOO in BAR
  Cleanup: Simplify boolean assignment
  Cleanup: Remove some unnecessary if (...) parens
  Fix invalid variable error for file-based disks
  FIX: Refactor DiagnoseOS to use a loop, not an inner fn
  FIX: Set INSTANCE_NICn_NETWORK_NAME only if net is defined
  StartInstance restores instance state if running
  Allow migrate --cleanup to adopt an instance
  Make migrate --cleanup more robust
  Make finalize_migration_{src,dst} a single op
  Make FinalizeMigration{Src,Dst} more robust
  Fix instance state detection in _Shutdowninstance
  Code cleanup in hypervisor backend
  Fix for incorrect parsing of DRBD versions
  Fix for instance reinstall not updating config
  Change a few errors to report names, not UUIDs
  Give atomicWriteFile temp filenames a more distinct pattern
  LV check failure should print instance name
  Add ganeti-noded and ganeti-rapi --max-clients options
  Disable logging CallRPCMethod timings in non-debug configs
  568 Update hv_kvm to handle output from qemu >= 1.6.0
  Improve cluster verify ssh key errors

Resolve merge conflicts:
  lib/cmdlib/common.py
    caused by 81a68a3 use names not uuids in warnings

  lib/rapi/testutils.py
    trivial module import tweaks

  lib/server/rapi.py
    caused by c1e18c4 --max-clients and pam options

  lib/storage/drbd_info.py
  test/py/ganeti.storage.drbd_unittest.py
    caused by 49980c6 DRBD version parsing -- fixed differently (and
    less flexibly) on master. Use 2.17's version

  test/hs/Test/Ganeti/OpCodes.hs
    caused by e763a23 improve test data generation touching code
    code in 2.16, and modify_ssh_setup cluster param

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoMerge branch 'stable-2.16' into stable-2.17 stable-2.17
Brian Foley [Fri, 16 Dec 2016 17:23:46 +0000 (17:23 +0000)]
Merge branch 'stable-2.16' into stable-2.17

* stable-2.15
  Fix gnt-instance console instance unpausing for xl toolstack
  Disable pylint too-many-nested-blocks in _RunCmdPipe
  Reduce nesting in import-export ProcessChildIO
  Reduce nesting in LUOobCommand.Exec
  Reduce nesting in LUInstanceCreate.RunOsScripts
  Reduce nesting in RemoveNodeSshKeyBulk key calculation
  Reduce nesting in RemoveNodeSshKeyBulk ssh logic
  Reduce nesting in gnt-cluster VerifyDisks missing disk loop
  Reduce nesting in _CheckVLANArguments
  Reduce nesting in StartDaemon
  Disable pylint bad-continuation warning
  Disable pylint superfluous-parens warning
  Disable pylint redefined-variable-type warning
  Disable pylint too-many-branches warnings
  Disable pylint broad-except warnings
  Disable incorrect pylint assigning-non-slot warning
  Quell pylint unbalanced-tuple-unpacking warning
  Cleanup: Use new-style classes everywhere
  Quell pylint socket.timeout warning
  Quell the pylint wrong-import-order warnings
  Quell cell-var-from-loop warning
  Use default value lambda param to avoid cell-var-from-loop
  Quell too-many-boolean-expressions
  Remove pylint tests removed in pylint 2.0
  Quell trailing newline
  Quell bad-whitespace warning
  Quell consider-using-enumerate warning
  Disable pylint unsubscriptable-object warning
  Disable pylint bare-except warning
  Disable unwanted pylint wrong-import-position warnings
  Disable pylint unused-wildcard-import warning
  Disable incorrect pylint not-callable warning
  Disable pylint unpacking-non-sequence warning
  Disable pylint misplaced-comparison-constant warning
  Disable incorect pylint simplify-if-statement warning
  Disable pylint eval-used warning
  Disable pylint invalid-name warning
  Disable pylint import-self warning
  Disable some pylint unused-import warnings
  Replace deprecated pylint >=0.27 pragma with new form
  Delete old warning disables removed from pylint 1.6
  Fix pylint >1.4 pycurl no-member warnings
  Cleanup: Remove unused/duplicate module/fn import
  Cleanup: Fix unidiomatic-typecheck
  Cleanup: Remove some unneeded pylint disables
  Cleanup: Iterate dict rather than key list
  Cleanup: Remove unused format key
  Cleanup: StartInstance and RebootInstance return None
  Cleanup: Fix for/else with no break in AddAuthorizedKeys
  Cleanup: Replace map/filters with list comprehensions
  Cleanup: del is a statement not a function
  Cleanup: Use FOO not in BAR instead of not FOO in BAR
  Cleanup: Simplify boolean assignment
  Cleanup: Remove some unnecessary if (...) parens
  Fix invalid variable error for file-based disks
  FIX: Refactor DiagnoseOS to use a loop, not an inner fn
  FIX: Set INSTANCE_NICn_NETWORK_NAME only if net is defined
  StartInstance restores instance state if running
  Allow migrate --cleanup to adopt an instance
  Make migrate --cleanup more robust
  Make finalize_migration_{src,dst} a single op
  Make FinalizeMigration{Src,Dst} more robust
  Fix instance state detection in _Shutdowninstance
  Code cleanup in hypervisor backend
  Fix for incorrect parsing of DRBD versions
  Fix for instance reinstall not updating config
  Change a few errors to report names, not UUIDs
  Give atomicWriteFile temp filenames a more distinct pattern
  LV check failure should print instance name
  Add ganeti-noded and ganeti-rapi --max-clients options
  Disable logging CallRPCMethod timings in non-debug configs
  568 Update hv_kvm to handle output from qemu >= 1.6.0
  Improve cluster verify ssh key errors
  Fix inconsistent spaces vs tabs indent in makefile

* stable-2.13
  Bugfix: migrate needs HypervisorClass, not an instance

Fix easy merge conflict in lib/backend.py -- dead code removed in 2.15

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoMerge branch 'stable-2.15' into stable-2.16
Brian Foley [Fri, 16 Dec 2016 17:12:37 +0000 (17:12 +0000)]
Merge branch 'stable-2.15' into stable-2.16

Merge forward mostly cleanup patches from stable-2.15

* stable-2.15
  Fix gnt-instance console instance unpausing for xl toolstack
  Disable pylint too-many-nested-blocks in _RunCmdPipe
  Reduce nesting in import-export ProcessChildIO
  Reduce nesting in LUOobCommand.Exec
  Reduce nesting in LUInstanceCreate.RunOsScripts
  Reduce nesting in RemoveNodeSshKeyBulk key calculation
  Reduce nesting in RemoveNodeSshKeyBulk ssh logic
  Reduce nesting in gnt-cluster VerifyDisks missing disk loop
  Reduce nesting in _CheckVLANArguments
  Reduce nesting in StartDaemon
  Disable pylint bad-continuation warning
  Disable pylint superfluous-parens warning
  Disable pylint redefined-variable-type warning
  Disable pylint too-many-branches warnings
  Disable pylint broad-except warnings
  Disable incorrect pylint assigning-non-slot warning
  Quell pylint unbalanced-tuple-unpacking warning
  Cleanup: Use new-style classes everywhere
  Quell pylint socket.timeout warning
  Quell the pylint wrong-import-order warnings
  Quell cell-var-from-loop warning
  Use default value lambda param to avoid cell-var-from-loop
  Quell too-many-boolean-expressions
  Remove pylint tests removed in pylint 2.0
  Quell trailing newline
  Quell bad-whitespace warning
  Quell consider-using-enumerate warning
  Disable pylint unsubscriptable-object warning
  Disable pylint bare-except warning
  Disable unwanted pylint wrong-import-position warnings
  Disable pylint unused-wildcard-import warning
  Disable incorrect pylint not-callable warning
  Disable pylint unpacking-non-sequence warning
  Disable pylint misplaced-comparison-constant warning
  Disable incorect pylint simplify-if-statement warning
  Disable pylint eval-used warning
  Disable pylint invalid-name warning
  Disable pylint import-self warning
  Disable some pylint unused-import warnings
  Replace deprecated pylint >=0.27 pragma with new form
  Delete old warning disables removed from pylint 1.6
  Fix pylint >1.4 pycurl no-member warnings
  Cleanup: Remove unused/duplicate module/fn import
  Cleanup: Fix unidiomatic-typecheck
  Cleanup: Remove some unneeded pylint disables
  Cleanup: Iterate dict rather than key list
  Cleanup: Remove unused format key
  Cleanup: StartInstance and RebootInstance return None
  Cleanup: Fix for/else with no break in AddAuthorizedKeys
  Cleanup: Replace map/filters with list comprehensions
  Cleanup: del is a statement not a function
  Cleanup: Use FOO not in BAR instead of not FOO in BAR
  Cleanup: Simplify boolean assignment
  Cleanup: Remove some unnecessary if (...) parens
  Fix invalid variable error for file-based disks
  FIX: Refactor DiagnoseOS to use a loop, not an inner fn
  FIX: Set INSTANCE_NICn_NETWORK_NAME only if net is defined
  Fix for incorrect parsing of DRBD versions
  Change a few errors to report names, not UUIDs
  Give atomicWriteFile temp filenames a more distinct pattern
  LV check failure should print instance name
  Disable logging CallRPCMethod timings in non-debug configs
  568 Update hv_kvm to handle output from qemu >= 1.6.0

Trivial merge conflicts:
  lib/cli.py                   -- whitespace
  lib/cmdlib/cluster/verify.py -- code previously removed from 2.16
  lib/hypervisor/hv_xen.py     -- tuple vs list
  lib/luxi.py                  -- whitespace
  lib/server/masterd.py        -- code cleaned up in 2.15 and
                                  previously removed in 2.16
  lib/storage/filestorage.py   -- whitespace & generator instead of list
  lib/tools/node_cleanup.py    -- whitespace

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoImprove the test data generation algorithms
Federico Morg Pareschi [Fri, 16 Dec 2016 15:00:41 +0000 (15:00 +0000)]
Improve the test data generation algorithms

This commit improves some of the test data generation in the ConfigData
dummy values of quickcheck. It generates more realistic instance, node
and group names by sampling them from the provided ConfigData. This
makes it possible to test for consistent data validation that might
depend on multiple names being present in the cluster configuration at
the same time.

Signed-off-by: Federico Morg Pareschi <morg@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoImplement starvation-prevention mechanism in queue
Federico Morg Pareschi [Fri, 16 Dec 2016 13:51:41 +0000 (13:51 +0000)]
Implement starvation-prevention mechanism in queue

This patch adds a starvation prevention mechanism to the Ganeti
predictive job queue. It calculates the age of submitted jobs
comparing the current time to the submission time and appropriately
adjusts the job's lock weight accordingly to avoid starvation.

Signed-off-by: Federico Morg Pareschi <morg@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoFix gnt-instance console instance unpausing for xl toolstack
Nicolas Avrutin [Thu, 15 Dec 2016 17:05:46 +0000 (09:05 -0800)]
Fix gnt-instance console instance unpausing for xl toolstack

xen-console-wrapper called the "xm" command instead of using the toolstack
configured in Ganeti. This patch replaces those calls with calls to the
toolstack command which is passed as the first argument to the script.

In addition, whereas "xm list --long" output a sexp, "xl list --long" outputs
JSON. Unfortunately, the JSON data is missing the "state" field, which we need
to determine whether the instance is paused. Thus we have to use
"xm list"/"xl list" and chop off the header line (because there is no flag
to omit printing the headers).

Signed-off-by: Nicolas Avrutin <rasputin@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoDisable pylint too-many-nested-blocks in _RunCmdPipe
Brian Foley [Tue, 6 Dec 2016 14:46:32 +0000 (14:46 +0000)]
Disable pylint too-many-nested-blocks in _RunCmdPipe

There doesn't appear to be any easy way of reducing the complexity
of this function without moving it elsewhere or completely reorganising
the function, so disable this warning for the time being.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoReduce nesting in import-export ProcessChildIO
Brian Foley [Tue, 6 Dec 2016 15:18:06 +0000 (15:18 +0000)]
Reduce nesting in import-export ProcessChildIO

This avoids some pylint too-many-nested-blocks warnings. Do this by
flattening some 'arrow antipattern code' inside the polling loop to use
guards instead.

This is hard to see from diff, but git difftool -y -x 'diff -y' will
show the before and after side by side.

Sprinkle some comments while we're here.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoReduce nesting in LUOobCommand.Exec
Brian Foley [Tue, 6 Dec 2016 15:22:24 +0000 (15:22 +0000)]
Reduce nesting in LUOobCommand.Exec

This avoids a pylint too-many-nested-blocks warning.

NB it's hard to see from the diff because of all the whitespace, but
this just turns the if result.fail_msg check into a guard that continues
the next iteration of the loop, and dedends the rest of the loop body
rather than having it as an else:

git diff -w will show this.

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoReduce nesting in LUInstanceCreate.RunOsScripts
Brian Foley [Tue, 6 Dec 2016 15:21:59 +0000 (15:21 +0000)]
Reduce nesting in LUInstanceCreate.RunOsScripts

This avoids a pylint too-many-nested-blocks warning.

NB it's hard to see from the diff because of all the whitespace, but
this just replaces the top "if iobj.disk and not self.adopt_disks" with
a pair of guards that return early, and dedents the rest of the fn.

git diff -w will show this.

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoReduce nesting in RemoveNodeSshKeyBulk key calculation
Brian Foley [Tue, 6 Dec 2016 14:30:57 +0000 (14:30 +0000)]
Reduce nesting in RemoveNodeSshKeyBulk key calculation

This avoids a pylint too-many-nested-blocks warning. This removes the
6th level of nesting in the function, but may also be marginally faster
by turning the calculation into the set difference operation it really
is.

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoReduce nesting in RemoveNodeSshKeyBulk ssh logic
Brian Foley [Tue, 6 Dec 2016 14:15:14 +0000 (14:15 +0000)]
Reduce nesting in RemoveNodeSshKeyBulk ssh logic

This avoids a pylint too-many-nested-blocks warning. It also removes
some copy & paste code, showing that the master candidate and ordinary
node case are the same apart from the logging.

No funtional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoReduce nesting in gnt-cluster VerifyDisks missing disk loop
Brian Foley [Tue, 6 Dec 2016 15:19:01 +0000 (15:19 +0000)]
Reduce nesting in gnt-cluster VerifyDisks missing disk loop

To avoid pylint's too-many-nested-blocks warning.

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoReduce nesting in _CheckVLANArguments
Brian Foley [Tue, 6 Dec 2016 15:19:50 +0000 (15:19 +0000)]
Reduce nesting in _CheckVLANArguments

This avoids a pylint too-many-nested-blocks warning. It also has the
happy side effect of removing some duplicated code.

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoReduce nesting in StartDaemon
Brian Foley [Tue, 6 Dec 2016 15:22:36 +0000 (15:22 +0000)]
Reduce nesting in StartDaemon

This avoids a pylint too-many-nested-blocks warning.

The extra try: finally: os._exit(1) is unnecessary as _StartDaemonChild
already catches all its exceptions and if it ever finishes, calls
os._exit(1) anyways.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoDisable pylint bad-continuation warning
Brian Foley [Mon, 5 Dec 2016 18:41:28 +0000 (18:41 +0000)]
Disable pylint bad-continuation warning

pylint is much more strict than pep8, and it would be too invasive
(and arguably pointless) to update these right now.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoDisable pylint superfluous-parens warning
Brian Foley [Mon, 5 Dec 2016 18:20:13 +0000 (18:20 +0000)]
Disable pylint superfluous-parens warning

There are too many cases where we deliberately wrap expressions in
parens, either to indicate comparisons, or to allow multiline
expressions without line continuation chars, or to clarify confusing
precedence.

While here, clean up a few unpythonic cases.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoDisable pylint redefined-variable-type warning
Brian Foley [Mon, 5 Dec 2016 17:29:30 +0000 (17:29 +0000)]
Disable pylint redefined-variable-type warning

There are a large number of cases where Ganeti assigns multiple types
(eg set/list, float/int) to the same variable at different times, and
although these would make a type checking tool very unhappy, they are
benign here (and besides, cleaning all this up would be too invasive).

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoDisable pylint too-many-branches warnings
Brian Foley [Mon, 5 Dec 2016 11:39:36 +0000 (11:39 +0000)]
Disable pylint too-many-branches warnings

This is useful, but in some cases is a little too conservative. A fn
can have a lot of branches, but very little nesting, and can still be
easy to understand. This occurs in, eg, XenPvmHypervisor._GetConfig.

Ganeti has many functions that fail this check, so disable it for now
until we get a chance to clean up the codebase.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoDisable pylint broad-except warnings
Brian Foley [Sun, 4 Dec 2016 21:01:56 +0000 (21:01 +0000)]
Disable pylint broad-except warnings

These are all deliberate top-level catch-any-unhandled-exception cases,
used for logging and error handling so just get pylint to ignore them.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoDisable incorrect pylint assigning-non-slot warning
Brian Foley [Mon, 5 Dec 2016 11:07:27 +0000 (11:07 +0000)]
Disable incorrect pylint assigning-non-slot warning

This occurs pretty heavily in lib/objects.py, where pylint isn't
correctly detecting the __slots__ assignment. This appears to be
a known issue: https://github.com/PyCQA/pylint/issues/379

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoQuell pylint unbalanced-tuple-unpacking warning
Brian Foley [Sun, 4 Dec 2016 20:55:26 +0000 (20:55 +0000)]
Quell pylint unbalanced-tuple-unpacking warning

Both of these functions return a list, not a tuple, and by manually
tracing the code, we can see they always return non-empty lists.

Change the callers to get the first element of the list rather than
using destructuring.

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoCleanup: Use new-style classes everywhere
Brian Foley [Mon, 5 Dec 2016 16:56:58 +0000 (16:56 +0000)]
Cleanup: Use new-style classes everywhere

This quells pylint's old-style-class warning. For all classes changed by
this commit, none of the differences between new-style and old classes
matter: we don't subclass any of these classes, or use super()/type().

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoQuell pylint socket.timeout warning
Brian Foley [Sun, 4 Dec 2016 19:12:53 +0000 (19:12 +0000)]
Quell pylint socket.timeout warning

socket.timeout doesn't have string or errno attributes.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoQuell the pylint wrong-import-order warnings
Brian Foley [Tue, 18 Oct 2016 19:46:38 +0000 (20:46 +0100)]
Quell the pylint wrong-import-order warnings

Sort the imports to what pylint thinks is correct.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoQuell cell-var-from-loop warning
Brian Foley [Tue, 18 Oct 2016 19:15:49 +0000 (20:15 +0100)]
Quell cell-var-from-loop warning

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoUse default value lambda param to avoid cell-var-from-loop
Brian Foley [Tue, 18 Oct 2016 19:11:04 +0000 (20:11 +0100)]
Use default value lambda param to avoid cell-var-from-loop

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoQuell too-many-boolean-expressions
Brian Foley [Tue, 18 Oct 2016 18:40:28 +0000 (19:40 +0100)]
Quell too-many-boolean-expressions

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoRemove pylint tests removed in pylint 2.0
Brian Foley [Tue, 18 Oct 2016 17:31:43 +0000 (18:31 +0100)]
Remove pylint tests removed in pylint 2.0

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoQuell trailing newline
Brian Foley [Tue, 18 Oct 2016 17:12:57 +0000 (18:12 +0100)]
Quell trailing newline

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoQuell bad-whitespace warning
Brian Foley [Mon, 17 Oct 2016 15:19:09 +0000 (16:19 +0100)]
Quell bad-whitespace warning

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoQuell consider-using-enumerate warning
Brian Foley [Mon, 17 Oct 2016 15:05:07 +0000 (16:05 +0100)]
Quell consider-using-enumerate warning

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable pylint unsubscriptable-object warning
Brian Foley [Sun, 4 Dec 2016 20:42:41 +0000 (20:42 +0000)]
Disable pylint unsubscriptable-object warning

In these cases result is subscriptable.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable pylint bare-except warning
Brian Foley [Sun, 4 Dec 2016 19:20:58 +0000 (19:20 +0000)]
Disable pylint bare-except warning

This is actually dead code which wasn't removed until Ganeti 2.16

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable unwanted pylint wrong-import-position warnings
Brian Foley [Tue, 18 Oct 2016 19:51:41 +0000 (20:51 +0100)]
Disable unwanted pylint wrong-import-position warnings

For these, we want to import some modules in a particular order, so
disable the pylint warnings.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable pylint unused-wildcard-import warning
Brian Foley [Tue, 18 Oct 2016 19:56:57 +0000 (20:56 +0100)]
Disable pylint unused-wildcard-import warning

We're deliberately importing all the variables defined in cli_opt into
cli's namespace.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable incorrect pylint not-callable warning
Brian Foley [Tue, 18 Oct 2016 19:23:03 +0000 (20:23 +0100)]
Disable incorrect pylint not-callable warning

REQ_REQUEST will be a parameter validating function from ht, but pylint
can't statically determine this.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable pylint unpacking-non-sequence warning
Brian Foley [Tue, 18 Oct 2016 19:21:46 +0000 (20:21 +0100)]
Disable pylint unpacking-non-sequence warning

For these cases, self.ia_result actually will have a tuple assigned to
it, it's just hard for static analysis tools to see it.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable pylint misplaced-comparison-constant warning
Brian Foley [Tue, 18 Oct 2016 19:20:12 +0000 (20:20 +0100)]
Disable pylint misplaced-comparison-constant warning

It's correct, and easier to read a < x < b in this case.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable incorect pylint simplify-if-statement warning
Brian Foley [Tue, 18 Oct 2016 19:01:04 +0000 (20:01 +0100)]
Disable incorect pylint simplify-if-statement warning

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable pylint eval-used warning
Brian Foley [Mon, 17 Oct 2016 15:00:03 +0000 (16:00 +0100)]
Disable pylint eval-used warning

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable pylint invalid-name warning
Brian Foley [Mon, 17 Oct 2016 14:07:56 +0000 (15:07 +0100)]
Disable pylint invalid-name warning

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable pylint import-self warning
Brian Foley [Mon, 17 Oct 2016 14:07:43 +0000 (15:07 +0100)]
Disable pylint import-self warning

In this case it's a deliberate test to make sure that the contents
of lib/ have been correctly installed in a dir called ganeti.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable some pylint unused-import warnings
Brian Foley [Mon, 17 Oct 2016 13:52:52 +0000 (14:52 +0100)]
Disable some pylint unused-import warnings

Technically the imports are unused in the module, but they're being
imported to make them available to users of the module.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoReplace deprecated pylint >=0.27 pragma with new form
Brian Foley [Sun, 4 Dec 2016 20:19:43 +0000 (20:19 +0000)]
Replace deprecated pylint >=0.27 pragma with new form

The disable-all pragma was deprecated in favour of the new skip-file
pragma in pylint >= 0.27. Since we're moving to 1.6.4, use this instead.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDelete old warning disables removed from pylint 1.6
Brian Foley [Tue, 18 Oct 2016 20:37:33 +0000 (21:37 +0100)]
Delete old warning disables removed from pylint 1.6

These include star-args and r0924: badly implemented immutable.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoFix pylint >1.4 pycurl no-member warnings
Brian Foley [Sun, 4 Dec 2016 18:49:16 +0000 (18:49 +0000)]
Fix pylint >1.4 pycurl no-member warnings

Add pycurl to the whitelist of C extensions that can be loaded for the
purposes of attribute checking. This works around a security feature
introduced in pylint 1.4 to avoid arbitrary code injection.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: Remove unused/duplicate module/fn import
Brian Foley [Tue, 18 Oct 2016 18:33:39 +0000 (19:33 +0100)]
Cleanup: Remove unused/duplicate module/fn import

No functional change. Found by pylint's reimported warning.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: Fix unidiomatic-typecheck
Brian Foley [Tue, 18 Oct 2016 17:29:15 +0000 (18:29 +0100)]
Cleanup: Fix unidiomatic-typecheck

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: Remove some unneeded pylint disables
Brian Foley [Tue, 18 Oct 2016 17:22:52 +0000 (18:22 +0100)]
Cleanup: Remove some unneeded pylint disables

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: Iterate dict rather than key list
Brian Foley [Mon, 17 Oct 2016 16:49:50 +0000 (17:49 +0100)]
Cleanup: Iterate dict rather than key list

The later explicitly constructs a new list of the dictionary's keys, which
we don't need here. Quietens pylint's consider-iterating-dictionary warning.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: Remove unused format key
Brian Foley [Mon, 17 Oct 2016 15:48:03 +0000 (16:48 +0100)]
Cleanup: Remove unused format key

No functional change, but quietens pylint unused-format-string-argument
warning.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: StartInstance and RebootInstance return None
Brian Foley [Mon, 17 Oct 2016 15:22:14 +0000 (16:22 +0100)]
Cleanup: StartInstance and RebootInstance return None

Quell assignment-from-none warning

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: Fix for/else with no break in AddAuthorizedKeys
Brian Foley [Sun, 4 Dec 2016 19:35:09 +0000 (19:35 +0000)]
Cleanup: Fix for/else with no break in AddAuthorizedKeys

After a refactoring in c178396, the only break statement in the for/else
loop in AddAuthorizedKeys was removed. This means the else block is
always executed, so remove the else: to quell a pylint warning.

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: Replace map/filters with list comprehensions
Brian Foley [Mon, 17 Oct 2016 14:54:27 +0000 (15:54 +0100)]
Cleanup: Replace map/filters with list comprehensions

No functional change, but more pythonic and quells pylint's
deprecated-lambda warning.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: del is a statement not a function
Brian Foley [Mon, 17 Oct 2016 13:16:40 +0000 (14:16 +0100)]
Cleanup: del is a statement not a function

Remove unnecessary parens to make pylint happy.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: Use FOO not in BAR instead of not FOO in BAR
Brian Foley [Mon, 17 Oct 2016 13:14:31 +0000 (14:14 +0100)]
Cleanup: Use FOO not in BAR instead of not FOO in BAR

No functional change, but more pythonic, and makes pylint's
unneeded-not warning happy.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: Simplify boolean assignment
Brian Foley [Fri, 2 Dec 2016 17:14:32 +0000 (17:14 +0000)]
Cleanup: Simplify boolean assignment

No functional change. Found by pylint's simplifiable-if-statement.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: Remove some unnecessary if (...) parens
Brian Foley [Fri, 2 Dec 2016 17:11:02 +0000 (17:11 +0000)]
Cleanup: Remove some unnecessary if (...) parens

No functional change, but makes pylint 1.1.0 a little happier, and
is more pythonic.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoAdd OS params removal in gnt-instance reinstall
Yiannis Tsiouris [Mon, 5 Dec 2016 14:41:39 +0000 (16:41 +0200)]
Add OS params removal in gnt-instance reinstall

This patch adds '--remove-os-parameters' and
'--remove-os-parameters-private' options to 'gnt-instance reinstall'.
Similarly to 'gnt-instance modify', the new options can be used to
perform reinstalls to OS providers that support different parameters.
E.g.:

$ gnt-instance reinstall --remove-os-parameters param1,param2 \
  -o <new_os_provider> -O param3=val3 <instance_name>

or

$ gnt-instance reinstall --remove-os-parameters param1,param2 \
  -o <new_os_provider> -O param3=val3 <instance_name>

Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoSupport clear OS params in gnt-instance reinstall
Yiannis Tsiouris [Mon, 5 Dec 2016 14:41:38 +0000 (16:41 +0200)]
Support clear OS params in gnt-instance reinstall

This extends 'gnt-instance reinstall' to support
'--clear-os-parameters' and '--clear-os-parameters-private' options from
'gnt-instance modify'. E.g.:

$ gnt-instance reinstall --clear-os-parameters -o <new_os_provider> \
  -O param1=val1,param2=val2 <instance_name>

or

$ gnt-instance reinstall --clear-os-parameters-private \
  -o <new_os_provider> -O param1=val1,param2=val2 <instance_name>

Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoTest new options of gnt-instance modify in RAPI
Yiannis Tsiouris [Mon, 5 Dec 2016 14:41:37 +0000 (16:41 +0200)]
Test new options of gnt-instance modify in RAPI

This patch adds tests to RAPI for gnt-instance modify options:
- '--clear-os-parameters'/'--clear-os-parameters-private'
- '--remove-os-parameters'/'--remove-os-parameters-private'

Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoAdd cmdlib tests for removing instance OS params
Nikos Skalkotos [Mon, 5 Dec 2016 14:41:36 +0000 (16:41 +0200)]
Add cmdlib tests for removing instance OS params

Add LUInstanceSetParams tests for removing individual public & private
parameters, as well as, for clearing out public & private parameters.

Signed-off-by: Nikos Skalkotos <skalkoto@grnet.gr>
Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoSupport OS params removal in gnt-instance modify
Yiannis Tsiouris [Mon, 5 Dec 2016 14:41:35 +0000 (16:41 +0200)]
Support OS params removal in gnt-instance modify

This patch extends 'gnt-instance modify' by allowing a user to remove a
list of public/private OS parameters from an instance. This can be
useful before performing a reinstall to a new OS provider. Example
usage:

$ gnt-instance modify --remove-os-parameters parm1,parm2 <instance_name>

or

$ gnt-instance modify --remove-os-parameters-private parm3 <instance_name>

Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoAdd clear OS params options in gnt-instance modify
Yiannis Tsiouris [Mon, 5 Dec 2016 14:41:34 +0000 (16:41 +0200)]
Add clear OS params options in gnt-instance modify

This patch adds two new options in 'gnt-instance modify' which allow the
user to clear all current public/private OS parameters of an instance.
This might be useful for OS providers that consider no parameters as
valid or, more commonly, for changing OS provider (and parameters)
before performing a reinstall. E.g.:

$ gnt-instance modify --clear-os-parameters <instance_name>

or

$ gnt-instance modify --clear-os-parameters-private <instance_name>

Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoFix invalid variable error for file-based disks
Brian Foley [Mon, 17 Oct 2016 16:27:26 +0000 (17:27 +0100)]
Fix invalid variable error for file-based disks

Introduced by 94e252a33. Found by pylint's undefined-loop-variable
warning.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoFIX: Refactor DiagnoseOS to use a loop, not an inner fn
Brian Foley [Mon, 17 Oct 2016 13:24:26 +0000 (14:24 +0100)]
FIX: Refactor DiagnoseOS to use a loop, not an inner fn

Found by pylint's cell-var-from-loop.

The inner function tries to access a dictionary that's local to the
outer function, assuming that Python has similar lexical scoping rules
to most languages. Unfortunately it doesn't. See
https://docs.python.org/3/tutorial/classes.html#python-scopes-and-namespaces

Refactor code to work as intended by moving the fn body into a loop.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

2 years agoFIX: Set INSTANCE_NICn_NETWORK_NAME only if net is defined
Brian Foley [Mon, 17 Oct 2016 13:08:32 +0000 (14:08 +0100)]
FIX: Set INSTANCE_NICn_NETWORK_NAME only if net is defined

Found by pylint's using-constant-test warning. 'network' is a module,
'net' is the network name for this NIC.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

2 years agoStartInstance restores instance state if running
Viktor Bachraty [Fri, 7 Oct 2016 19:10:34 +0000 (20:10 +0100)]
StartInstance restores instance state if running

Add a side effect to StartInstance on Xen hypervisor that restores the
Xen configuration in case it is missing. This is a common failure
scenario during migrations. The generic backend checks if the instance
is running and queries the hypervisor about it's state. If it needs
recovery, the backend gathers the associated block devices and calls
RestoreInstance in the same manner as if it would call StartInstance in
case the instance wasn't already running.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoAllow migrate --cleanup to adopt an instance
Viktor Bachraty [Fri, 7 Oct 2016 17:57:22 +0000 (18:57 +0100)]
Allow migrate --cleanup to adopt an instance

In some failure modes, Ganeti state of record may desync from actual
state of world. This patch allows migrate --cleanup to adopt an instance
if it is detected running on an unexpected node.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoMake migrate --cleanup more robust
Viktor Bachraty [Fri, 7 Oct 2016 16:29:23 +0000 (17:29 +0100)]
Make migrate --cleanup more robust

In case of Xen migrations, the most common failure case is when the
instance fails to freeze so the migration fails with domains running on
both target and source node. This patchs allows migrate --cleanup to
recover by running AbortMigrate() in case the instance is running on
both the source and target node.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoMake finalize_migration_{src,dst} a single op
Viktor Bachraty [Fri, 7 Oct 2016 16:17:44 +0000 (17:17 +0100)]
Make finalize_migration_{src,dst} a single op

FinalizeMigrationSource and FinalizeMigrationDst should compose an
single operation consisting of 2 idempotent steps, that can be always
both retried, similarly how they are used in AbortMigration() without
any reliance on the order of their execution. Otherwise we would have to
remember at which step did the migration fail in order to be able to
rollback/recover. Also state of the record should be updated only if
both succeeded, otherwise we won't know in case of recovery if the
record corresponds to the original state or the final state.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoMake FinalizeMigration{Src,Dst} more robust
Viktor Bachraty [Fri, 7 Oct 2016 16:12:23 +0000 (17:12 +0100)]
Make FinalizeMigration{Src,Dst} more robust

Migration is supported on KVM/Xen hypervisors only. Finalize methods are
called in both success or suspected failure (the master node can't
certainly know about failures - e.g. in case of timeouts). Thus these
functions need to be as robust as possible in order to be able to
recover from failures - in case success=False the functions should be
conservative, trying to detect the actual state and idempotent. The
cleanup workaround works for XM, on XL it is a no-op.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoFix instance state detection in _Shutdowninstance
Viktor Bachraty [Thu, 6 Oct 2016 18:52:49 +0000 (19:52 +0100)]
Fix instance state detection in _Shutdowninstance

GetInstanceInfo returns info that is already parsed with state being an
enum value converted from string using _XenToHypervisorInstanceState()

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoCode cleanup in hypervisor backend
Viktor Bachraty [Thu, 6 Oct 2016 18:34:51 +0000 (19:34 +0100)]
Code cleanup in hypervisor backend

Non-functional changes, smaller code cleanup.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoFix for incorrect parsing of DRBD versions
Federico Morg Pareschi [Fri, 2 Dec 2016 11:12:25 +0000 (11:12 +0000)]
Fix for incorrect parsing of DRBD versions

Following issue #1194, this patch allows Ganeti to correctly
parse drbd versions that also include a dash in their k-fix
version component.

This technically makes Ganeti treat x.y.z.k the same as x.y.z-k
with regards to compatibility. For practicality we can assume that
these versions are the same, this is the intended behaviour as it is
unlikely that these two versions would be incompatible with each
other, if that ever were to happen.

Signed-off-by: Federico Morg Pareschi <morg@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoFix for instance reinstall not updating config
Federico Morg Pareschi [Mon, 28 Nov 2016 13:58:06 +0000 (13:58 +0000)]
Fix for instance reinstall not updating config

As per issue #1193, instance reinstalls with modified OS parameters
did not persist among reinstalls and were not saved to the cluster's
config data. With this patch, this problem should be fixed.

Signed-off-by: Federico Morg Pareschi <morg@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

2 years agoChange a few errors to report names, not UUIDs
Brian Foley [Thu, 29 Sep 2016 13:04:35 +0000 (14:04 +0100)]
Change a few errors to report names, not UUIDs

Node/Node group name is much easier for humans to read.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

2 years agoGive atomicWriteFile temp filenames a more distinct pattern
Brian Foley [Wed, 28 Sep 2016 13:25:49 +0000 (14:25 +0100)]
Give atomicWriteFile temp filenames a more distinct pattern

This stops them being confused with 'real' filenames by both users and
code if the daemon crashes before the temporary file is removed.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoFirst implementation of the predictive scheduler
Federico Morg Pareschi [Thu, 15 Sep 2016 12:55:33 +0000 (13:55 +0100)]
First implementation of the predictive scheduler

This is just a partial implementation of the static lock prediction system.
With this, Ganeti queue system tries to predict the locks of the running
jobs and schedule new jobs that are less likely to block.
It is a partial implementation which might cause starvation for jobs stuck
in the queue, following patches will address this issue.

Signed-off-by: Federico Morg Pareschi <morg@google.com>
Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

2 years agoLV check failure should print instance name
Viktor Bachraty [Wed, 14 Sep 2016 12:08:17 +0000 (13:08 +0100)]
LV check failure should print instance name

Instances with secondaries in a separate nodegroup from primary can
result in cluster verify throwing a missing node lock error for LV
check. This should properly print the node name and the violating
instance name.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoFix RAPI auth bug introduced by f3b51166
Brian Foley [Sun, 11 Sep 2016 21:20:48 +0000 (21:20 +0000)]
Fix RAPI auth bug introduced by f3b51166

This accidentally dropped the RemoteApiHandler.AuthenticationRequired
logic to check if a specific RAPI call handler required auth for
read/write access to the cluster config.

This wasn't noticed because some unit tests were changed to hardcode
auth to be required unconditionaly.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

2 years agoAdd ganeti-noded and ganeti-rapi --max-clients options
Brian Foley [Mon, 12 Sep 2016 14:52:49 +0000 (15:52 +0100)]
Add ganeti-noded and ganeti-rapi --max-clients options

This defaults to 20 -- the same as the current hardcoded limit.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>