ganeti.git
2 years agoExpose allow-failover option to RAPI client master
Yiannis Tsiouris [Mon, 6 Feb 2017 14:17:11 +0000 (16:17 +0200)]
Expose allow-failover option to RAPI client

'gnt-instance migrate' supports performing a failover on an instance
that is not possible to migrate (e.g. if the instance is down). This
patch exposes this option ('--allow-failover') to the RAPI client.

Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Signed-off-by: Alex Kiousis <alexk@noc.grnet.gr>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

2 years agoImplement predictive queue cluster parameter
Federico Morg Pareschi [Mon, 9 Jan 2017 11:17:58 +0000 (11:17 +0000)]
Implement predictive queue cluster parameter

This commit adds the enabled_predictive_queue cluster parameter that
allows the cluster to specify if the predictive scheduler should be used
or not when ordering ganeti jobs in the queue. It also adds the correct
command line flags for the cluster init and cluster modify ganeti jobs.

As a default, if no option is specified upon cluster creation, the
predictive scheduler is automatically enabled.

Signed-off-by: Federico Morg Pareschi <morg@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

2 years agoFix incorrect 'any' function call
Federico Morg Pareschi [Wed, 4 Jan 2017 15:11:41 +0000 (15:11 +0000)]
Fix incorrect 'any' function call

As introduced by commit 77807d83 and d388e30c6d
and was not caught during the code review process.

Signed-off-by: Federico Morg Pareschi <morg@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoFIX: Add missing LockDecls files to Makefile.am
Federico Morg Pareschi [Mon, 19 Dec 2016 16:03:15 +0000 (16:03 +0000)]
FIX: Add missing LockDecls files to Makefile.am

Trivial change to add the missing LockDecls.hs files to Makefile.am.
This should remove some complaints to some of the tests that failed to
build.

Signed-off-by: Federico Morg Pareschi <morg@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoQuell remaining pylint 1.6.4 warnings after merge
Brian Foley [Mon, 19 Dec 2016 14:19:09 +0000 (14:19 +0000)]
Quell remaining pylint 1.6.4 warnings after merge

These should all be harmless modifications with no functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoMerge branch 'stable-2.17'
Brian Foley [Mon, 19 Dec 2016 12:02:39 +0000 (12:02 +0000)]
Merge branch 'stable-2.17'

* stable-2.17
  1143: Add gnt-cluster modify --modify-ssh-setup option

* stable-2.15
  Fix gnt-instance console instance unpausing for xl toolstack
  Disable pylint too-many-nested-blocks in _RunCmdPipe
  Reduce nesting in import-export ProcessChildIO
  Reduce nesting in LUOobCommand.Exec
  Reduce nesting in LUInstanceCreate.RunOsScripts
  Reduce nesting in RemoveNodeSshKeyBulk key calculation
  Reduce nesting in RemoveNodeSshKeyBulk ssh logic
  Reduce nesting in gnt-cluster VerifyDisks missing disk loop
  Reduce nesting in _CheckVLANArguments
  Reduce nesting in StartDaemon
  Disable pylint bad-continuation warning
  Disable pylint superfluous-parens warning
  Disable pylint redefined-variable-type warning
  Disable pylint too-many-branches warnings
  Disable pylint broad-except warnings
  Disable incorrect pylint assigning-non-slot warning
  Quell pylint unbalanced-tuple-unpacking warning
  Cleanup: Use new-style classes everywhere
  Quell pylint socket.timeout warning
  Quell the pylint wrong-import-order warnings
  Quell cell-var-from-loop warning
  Use default value lambda param to avoid cell-var-from-loop
  Quell too-many-boolean-expressions
  Remove pylint tests removed in pylint 2.0
  Quell trailing newline
  Quell bad-whitespace warning
  Quell consider-using-enumerate warning
  Disable pylint unsubscriptable-object warning
  Disable pylint bare-except warning
  Disable unwanted pylint wrong-import-position warnings
  Disable pylint unused-wildcard-import warning
  Disable incorrect pylint not-callable warning
  Disable pylint unpacking-non-sequence warning
  Disable pylint misplaced-comparison-constant warning
  Disable incorect pylint simplify-if-statement warning
  Disable pylint eval-used warning
  Disable pylint invalid-name warning
  Disable pylint import-self warning
  Disable some pylint unused-import warnings
  Replace deprecated pylint >=0.27 pragma with new form
  Delete old warning disables removed from pylint 1.6
  Fix pylint >1.4 pycurl no-member warnings
  Cleanup: Remove unused/duplicate module/fn import
  Cleanup: Fix unidiomatic-typecheck
  Cleanup: Remove some unneeded pylint disables
  Cleanup: Iterate dict rather than key list
  Cleanup: Remove unused format key
  Cleanup: StartInstance and RebootInstance return None
  Cleanup: Fix for/else with no break in AddAuthorizedKeys
  Cleanup: Replace map/filters with list comprehensions
  Cleanup: del is a statement not a function
  Cleanup: Use FOO not in BAR instead of not FOO in BAR
  Cleanup: Simplify boolean assignment
  Cleanup: Remove some unnecessary if (...) parens
  Fix invalid variable error for file-based disks
  FIX: Refactor DiagnoseOS to use a loop, not an inner fn
  FIX: Set INSTANCE_NICn_NETWORK_NAME only if net is defined
  StartInstance restores instance state if running
  Allow migrate --cleanup to adopt an instance
  Make migrate --cleanup more robust
  Make finalize_migration_{src,dst} a single op
  Make FinalizeMigration{Src,Dst} more robust
  Fix instance state detection in _Shutdowninstance
  Code cleanup in hypervisor backend
  Fix for incorrect parsing of DRBD versions
  Fix for instance reinstall not updating config
  Change a few errors to report names, not UUIDs
  Give atomicWriteFile temp filenames a more distinct pattern
  LV check failure should print instance name
  Add ganeti-noded and ganeti-rapi --max-clients options
  Disable logging CallRPCMethod timings in non-debug configs
  568 Update hv_kvm to handle output from qemu >= 1.6.0
  Improve cluster verify ssh key errors

Resolve merge conflicts:
  lib/cmdlib/common.py
    caused by 81a68a3 use names not uuids in warnings

  lib/rapi/testutils.py
    trivial module import tweaks

  lib/server/rapi.py
    caused by c1e18c4 --max-clients and pam options

  lib/storage/drbd_info.py
  test/py/ganeti.storage.drbd_unittest.py
    caused by 49980c6 DRBD version parsing -- fixed differently (and
    less flexibly) on master. Use 2.17's version

  test/hs/Test/Ganeti/OpCodes.hs
    caused by e763a23 improve test data generation touching code
    code in 2.16, and modify_ssh_setup cluster param

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoMerge branch 'stable-2.16' into stable-2.17 stable-2.17
Brian Foley [Fri, 16 Dec 2016 17:23:46 +0000 (17:23 +0000)]
Merge branch 'stable-2.16' into stable-2.17

* stable-2.15
  Fix gnt-instance console instance unpausing for xl toolstack
  Disable pylint too-many-nested-blocks in _RunCmdPipe
  Reduce nesting in import-export ProcessChildIO
  Reduce nesting in LUOobCommand.Exec
  Reduce nesting in LUInstanceCreate.RunOsScripts
  Reduce nesting in RemoveNodeSshKeyBulk key calculation
  Reduce nesting in RemoveNodeSshKeyBulk ssh logic
  Reduce nesting in gnt-cluster VerifyDisks missing disk loop
  Reduce nesting in _CheckVLANArguments
  Reduce nesting in StartDaemon
  Disable pylint bad-continuation warning
  Disable pylint superfluous-parens warning
  Disable pylint redefined-variable-type warning
  Disable pylint too-many-branches warnings
  Disable pylint broad-except warnings
  Disable incorrect pylint assigning-non-slot warning
  Quell pylint unbalanced-tuple-unpacking warning
  Cleanup: Use new-style classes everywhere
  Quell pylint socket.timeout warning
  Quell the pylint wrong-import-order warnings
  Quell cell-var-from-loop warning
  Use default value lambda param to avoid cell-var-from-loop
  Quell too-many-boolean-expressions
  Remove pylint tests removed in pylint 2.0
  Quell trailing newline
  Quell bad-whitespace warning
  Quell consider-using-enumerate warning
  Disable pylint unsubscriptable-object warning
  Disable pylint bare-except warning
  Disable unwanted pylint wrong-import-position warnings
  Disable pylint unused-wildcard-import warning
  Disable incorrect pylint not-callable warning
  Disable pylint unpacking-non-sequence warning
  Disable pylint misplaced-comparison-constant warning
  Disable incorect pylint simplify-if-statement warning
  Disable pylint eval-used warning
  Disable pylint invalid-name warning
  Disable pylint import-self warning
  Disable some pylint unused-import warnings
  Replace deprecated pylint >=0.27 pragma with new form
  Delete old warning disables removed from pylint 1.6
  Fix pylint >1.4 pycurl no-member warnings
  Cleanup: Remove unused/duplicate module/fn import
  Cleanup: Fix unidiomatic-typecheck
  Cleanup: Remove some unneeded pylint disables
  Cleanup: Iterate dict rather than key list
  Cleanup: Remove unused format key
  Cleanup: StartInstance and RebootInstance return None
  Cleanup: Fix for/else with no break in AddAuthorizedKeys
  Cleanup: Replace map/filters with list comprehensions
  Cleanup: del is a statement not a function
  Cleanup: Use FOO not in BAR instead of not FOO in BAR
  Cleanup: Simplify boolean assignment
  Cleanup: Remove some unnecessary if (...) parens
  Fix invalid variable error for file-based disks
  FIX: Refactor DiagnoseOS to use a loop, not an inner fn
  FIX: Set INSTANCE_NICn_NETWORK_NAME only if net is defined
  StartInstance restores instance state if running
  Allow migrate --cleanup to adopt an instance
  Make migrate --cleanup more robust
  Make finalize_migration_{src,dst} a single op
  Make FinalizeMigration{Src,Dst} more robust
  Fix instance state detection in _Shutdowninstance
  Code cleanup in hypervisor backend
  Fix for incorrect parsing of DRBD versions
  Fix for instance reinstall not updating config
  Change a few errors to report names, not UUIDs
  Give atomicWriteFile temp filenames a more distinct pattern
  LV check failure should print instance name
  Add ganeti-noded and ganeti-rapi --max-clients options
  Disable logging CallRPCMethod timings in non-debug configs
  568 Update hv_kvm to handle output from qemu >= 1.6.0
  Improve cluster verify ssh key errors
  Fix inconsistent spaces vs tabs indent in makefile

* stable-2.13
  Bugfix: migrate needs HypervisorClass, not an instance

Fix easy merge conflict in lib/backend.py -- dead code removed in 2.15

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoMerge branch 'stable-2.15' into stable-2.16
Brian Foley [Fri, 16 Dec 2016 17:12:37 +0000 (17:12 +0000)]
Merge branch 'stable-2.15' into stable-2.16

Merge forward mostly cleanup patches from stable-2.15

* stable-2.15
  Fix gnt-instance console instance unpausing for xl toolstack
  Disable pylint too-many-nested-blocks in _RunCmdPipe
  Reduce nesting in import-export ProcessChildIO
  Reduce nesting in LUOobCommand.Exec
  Reduce nesting in LUInstanceCreate.RunOsScripts
  Reduce nesting in RemoveNodeSshKeyBulk key calculation
  Reduce nesting in RemoveNodeSshKeyBulk ssh logic
  Reduce nesting in gnt-cluster VerifyDisks missing disk loop
  Reduce nesting in _CheckVLANArguments
  Reduce nesting in StartDaemon
  Disable pylint bad-continuation warning
  Disable pylint superfluous-parens warning
  Disable pylint redefined-variable-type warning
  Disable pylint too-many-branches warnings
  Disable pylint broad-except warnings
  Disable incorrect pylint assigning-non-slot warning
  Quell pylint unbalanced-tuple-unpacking warning
  Cleanup: Use new-style classes everywhere
  Quell pylint socket.timeout warning
  Quell the pylint wrong-import-order warnings
  Quell cell-var-from-loop warning
  Use default value lambda param to avoid cell-var-from-loop
  Quell too-many-boolean-expressions
  Remove pylint tests removed in pylint 2.0
  Quell trailing newline
  Quell bad-whitespace warning
  Quell consider-using-enumerate warning
  Disable pylint unsubscriptable-object warning
  Disable pylint bare-except warning
  Disable unwanted pylint wrong-import-position warnings
  Disable pylint unused-wildcard-import warning
  Disable incorrect pylint not-callable warning
  Disable pylint unpacking-non-sequence warning
  Disable pylint misplaced-comparison-constant warning
  Disable incorect pylint simplify-if-statement warning
  Disable pylint eval-used warning
  Disable pylint invalid-name warning
  Disable pylint import-self warning
  Disable some pylint unused-import warnings
  Replace deprecated pylint >=0.27 pragma with new form
  Delete old warning disables removed from pylint 1.6
  Fix pylint >1.4 pycurl no-member warnings
  Cleanup: Remove unused/duplicate module/fn import
  Cleanup: Fix unidiomatic-typecheck
  Cleanup: Remove some unneeded pylint disables
  Cleanup: Iterate dict rather than key list
  Cleanup: Remove unused format key
  Cleanup: StartInstance and RebootInstance return None
  Cleanup: Fix for/else with no break in AddAuthorizedKeys
  Cleanup: Replace map/filters with list comprehensions
  Cleanup: del is a statement not a function
  Cleanup: Use FOO not in BAR instead of not FOO in BAR
  Cleanup: Simplify boolean assignment
  Cleanup: Remove some unnecessary if (...) parens
  Fix invalid variable error for file-based disks
  FIX: Refactor DiagnoseOS to use a loop, not an inner fn
  FIX: Set INSTANCE_NICn_NETWORK_NAME only if net is defined
  Fix for incorrect parsing of DRBD versions
  Change a few errors to report names, not UUIDs
  Give atomicWriteFile temp filenames a more distinct pattern
  LV check failure should print instance name
  Disable logging CallRPCMethod timings in non-debug configs
  568 Update hv_kvm to handle output from qemu >= 1.6.0

Trivial merge conflicts:
  lib/cli.py                   -- whitespace
  lib/cmdlib/cluster/verify.py -- code previously removed from 2.16
  lib/hypervisor/hv_xen.py     -- tuple vs list
  lib/luxi.py                  -- whitespace
  lib/server/masterd.py        -- code cleaned up in 2.15 and
                                  previously removed in 2.16
  lib/storage/filestorage.py   -- whitespace & generator instead of list
  lib/tools/node_cleanup.py    -- whitespace

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoImprove the test data generation algorithms
Federico Morg Pareschi [Fri, 16 Dec 2016 15:00:41 +0000 (15:00 +0000)]
Improve the test data generation algorithms

This commit improves some of the test data generation in the ConfigData
dummy values of quickcheck. It generates more realistic instance, node
and group names by sampling them from the provided ConfigData. This
makes it possible to test for consistent data validation that might
depend on multiple names being present in the cluster configuration at
the same time.

Signed-off-by: Federico Morg Pareschi <morg@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoImplement starvation-prevention mechanism in queue
Federico Morg Pareschi [Fri, 16 Dec 2016 13:51:41 +0000 (13:51 +0000)]
Implement starvation-prevention mechanism in queue

This patch adds a starvation prevention mechanism to the Ganeti
predictive job queue. It calculates the age of submitted jobs
comparing the current time to the submission time and appropriately
adjusts the job's lock weight accordingly to avoid starvation.

Signed-off-by: Federico Morg Pareschi <morg@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoFix gnt-instance console instance unpausing for xl toolstack
Nicolas Avrutin [Thu, 15 Dec 2016 17:05:46 +0000 (09:05 -0800)]
Fix gnt-instance console instance unpausing for xl toolstack

xen-console-wrapper called the "xm" command instead of using the toolstack
configured in Ganeti. This patch replaces those calls with calls to the
toolstack command which is passed as the first argument to the script.

In addition, whereas "xm list --long" output a sexp, "xl list --long" outputs
JSON. Unfortunately, the JSON data is missing the "state" field, which we need
to determine whether the instance is paused. Thus we have to use
"xm list"/"xl list" and chop off the header line (because there is no flag
to omit printing the headers).

Signed-off-by: Nicolas Avrutin <rasputin@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoDisable pylint too-many-nested-blocks in _RunCmdPipe
Brian Foley [Tue, 6 Dec 2016 14:46:32 +0000 (14:46 +0000)]
Disable pylint too-many-nested-blocks in _RunCmdPipe

There doesn't appear to be any easy way of reducing the complexity
of this function without moving it elsewhere or completely reorganising
the function, so disable this warning for the time being.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoReduce nesting in import-export ProcessChildIO
Brian Foley [Tue, 6 Dec 2016 15:18:06 +0000 (15:18 +0000)]
Reduce nesting in import-export ProcessChildIO

This avoids some pylint too-many-nested-blocks warnings. Do this by
flattening some 'arrow antipattern code' inside the polling loop to use
guards instead.

This is hard to see from diff, but git difftool -y -x 'diff -y' will
show the before and after side by side.

Sprinkle some comments while we're here.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoReduce nesting in LUOobCommand.Exec
Brian Foley [Tue, 6 Dec 2016 15:22:24 +0000 (15:22 +0000)]
Reduce nesting in LUOobCommand.Exec

This avoids a pylint too-many-nested-blocks warning.

NB it's hard to see from the diff because of all the whitespace, but
this just turns the if result.fail_msg check into a guard that continues
the next iteration of the loop, and dedends the rest of the loop body
rather than having it as an else:

git diff -w will show this.

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoReduce nesting in LUInstanceCreate.RunOsScripts
Brian Foley [Tue, 6 Dec 2016 15:21:59 +0000 (15:21 +0000)]
Reduce nesting in LUInstanceCreate.RunOsScripts

This avoids a pylint too-many-nested-blocks warning.

NB it's hard to see from the diff because of all the whitespace, but
this just replaces the top "if iobj.disk and not self.adopt_disks" with
a pair of guards that return early, and dedents the rest of the fn.

git diff -w will show this.

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoReduce nesting in RemoveNodeSshKeyBulk key calculation
Brian Foley [Tue, 6 Dec 2016 14:30:57 +0000 (14:30 +0000)]
Reduce nesting in RemoveNodeSshKeyBulk key calculation

This avoids a pylint too-many-nested-blocks warning. This removes the
6th level of nesting in the function, but may also be marginally faster
by turning the calculation into the set difference operation it really
is.

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoReduce nesting in RemoveNodeSshKeyBulk ssh logic
Brian Foley [Tue, 6 Dec 2016 14:15:14 +0000 (14:15 +0000)]
Reduce nesting in RemoveNodeSshKeyBulk ssh logic

This avoids a pylint too-many-nested-blocks warning. It also removes
some copy & paste code, showing that the master candidate and ordinary
node case are the same apart from the logging.

No funtional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoReduce nesting in gnt-cluster VerifyDisks missing disk loop
Brian Foley [Tue, 6 Dec 2016 15:19:01 +0000 (15:19 +0000)]
Reduce nesting in gnt-cluster VerifyDisks missing disk loop

To avoid pylint's too-many-nested-blocks warning.

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoReduce nesting in _CheckVLANArguments
Brian Foley [Tue, 6 Dec 2016 15:19:50 +0000 (15:19 +0000)]
Reduce nesting in _CheckVLANArguments

This avoids a pylint too-many-nested-blocks warning. It also has the
happy side effect of removing some duplicated code.

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoReduce nesting in StartDaemon
Brian Foley [Tue, 6 Dec 2016 15:22:36 +0000 (15:22 +0000)]
Reduce nesting in StartDaemon

This avoids a pylint too-many-nested-blocks warning.

The extra try: finally: os._exit(1) is unnecessary as _StartDaemonChild
already catches all its exceptions and if it ever finishes, calls
os._exit(1) anyways.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoDisable pylint bad-continuation warning
Brian Foley [Mon, 5 Dec 2016 18:41:28 +0000 (18:41 +0000)]
Disable pylint bad-continuation warning

pylint is much more strict than pep8, and it would be too invasive
(and arguably pointless) to update these right now.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoDisable pylint superfluous-parens warning
Brian Foley [Mon, 5 Dec 2016 18:20:13 +0000 (18:20 +0000)]
Disable pylint superfluous-parens warning

There are too many cases where we deliberately wrap expressions in
parens, either to indicate comparisons, or to allow multiline
expressions without line continuation chars, or to clarify confusing
precedence.

While here, clean up a few unpythonic cases.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoDisable pylint redefined-variable-type warning
Brian Foley [Mon, 5 Dec 2016 17:29:30 +0000 (17:29 +0000)]
Disable pylint redefined-variable-type warning

There are a large number of cases where Ganeti assigns multiple types
(eg set/list, float/int) to the same variable at different times, and
although these would make a type checking tool very unhappy, they are
benign here (and besides, cleaning all this up would be too invasive).

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoDisable pylint too-many-branches warnings
Brian Foley [Mon, 5 Dec 2016 11:39:36 +0000 (11:39 +0000)]
Disable pylint too-many-branches warnings

This is useful, but in some cases is a little too conservative. A fn
can have a lot of branches, but very little nesting, and can still be
easy to understand. This occurs in, eg, XenPvmHypervisor._GetConfig.

Ganeti has many functions that fail this check, so disable it for now
until we get a chance to clean up the codebase.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoDisable pylint broad-except warnings
Brian Foley [Sun, 4 Dec 2016 21:01:56 +0000 (21:01 +0000)]
Disable pylint broad-except warnings

These are all deliberate top-level catch-any-unhandled-exception cases,
used for logging and error handling so just get pylint to ignore them.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoDisable incorrect pylint assigning-non-slot warning
Brian Foley [Mon, 5 Dec 2016 11:07:27 +0000 (11:07 +0000)]
Disable incorrect pylint assigning-non-slot warning

This occurs pretty heavily in lib/objects.py, where pylint isn't
correctly detecting the __slots__ assignment. This appears to be
a known issue: https://github.com/PyCQA/pylint/issues/379

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoQuell pylint unbalanced-tuple-unpacking warning
Brian Foley [Sun, 4 Dec 2016 20:55:26 +0000 (20:55 +0000)]
Quell pylint unbalanced-tuple-unpacking warning

Both of these functions return a list, not a tuple, and by manually
tracing the code, we can see they always return non-empty lists.

Change the callers to get the first element of the list rather than
using destructuring.

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoCleanup: Use new-style classes everywhere
Brian Foley [Mon, 5 Dec 2016 16:56:58 +0000 (16:56 +0000)]
Cleanup: Use new-style classes everywhere

This quells pylint's old-style-class warning. For all classes changed by
this commit, none of the differences between new-style and old classes
matter: we don't subclass any of these classes, or use super()/type().

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoQuell pylint socket.timeout warning
Brian Foley [Sun, 4 Dec 2016 19:12:53 +0000 (19:12 +0000)]
Quell pylint socket.timeout warning

socket.timeout doesn't have string or errno attributes.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoQuell the pylint wrong-import-order warnings
Brian Foley [Tue, 18 Oct 2016 19:46:38 +0000 (20:46 +0100)]
Quell the pylint wrong-import-order warnings

Sort the imports to what pylint thinks is correct.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoQuell cell-var-from-loop warning
Brian Foley [Tue, 18 Oct 2016 19:15:49 +0000 (20:15 +0100)]
Quell cell-var-from-loop warning

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoUse default value lambda param to avoid cell-var-from-loop
Brian Foley [Tue, 18 Oct 2016 19:11:04 +0000 (20:11 +0100)]
Use default value lambda param to avoid cell-var-from-loop

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoQuell too-many-boolean-expressions
Brian Foley [Tue, 18 Oct 2016 18:40:28 +0000 (19:40 +0100)]
Quell too-many-boolean-expressions

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoRemove pylint tests removed in pylint 2.0
Brian Foley [Tue, 18 Oct 2016 17:31:43 +0000 (18:31 +0100)]
Remove pylint tests removed in pylint 2.0

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoQuell trailing newline
Brian Foley [Tue, 18 Oct 2016 17:12:57 +0000 (18:12 +0100)]
Quell trailing newline

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoQuell bad-whitespace warning
Brian Foley [Mon, 17 Oct 2016 15:19:09 +0000 (16:19 +0100)]
Quell bad-whitespace warning

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoQuell consider-using-enumerate warning
Brian Foley [Mon, 17 Oct 2016 15:05:07 +0000 (16:05 +0100)]
Quell consider-using-enumerate warning

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable pylint unsubscriptable-object warning
Brian Foley [Sun, 4 Dec 2016 20:42:41 +0000 (20:42 +0000)]
Disable pylint unsubscriptable-object warning

In these cases result is subscriptable.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable pylint bare-except warning
Brian Foley [Sun, 4 Dec 2016 19:20:58 +0000 (19:20 +0000)]
Disable pylint bare-except warning

This is actually dead code which wasn't removed until Ganeti 2.16

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable unwanted pylint wrong-import-position warnings
Brian Foley [Tue, 18 Oct 2016 19:51:41 +0000 (20:51 +0100)]
Disable unwanted pylint wrong-import-position warnings

For these, we want to import some modules in a particular order, so
disable the pylint warnings.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable pylint unused-wildcard-import warning
Brian Foley [Tue, 18 Oct 2016 19:56:57 +0000 (20:56 +0100)]
Disable pylint unused-wildcard-import warning

We're deliberately importing all the variables defined in cli_opt into
cli's namespace.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable incorrect pylint not-callable warning
Brian Foley [Tue, 18 Oct 2016 19:23:03 +0000 (20:23 +0100)]
Disable incorrect pylint not-callable warning

REQ_REQUEST will be a parameter validating function from ht, but pylint
can't statically determine this.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable pylint unpacking-non-sequence warning
Brian Foley [Tue, 18 Oct 2016 19:21:46 +0000 (20:21 +0100)]
Disable pylint unpacking-non-sequence warning

For these cases, self.ia_result actually will have a tuple assigned to
it, it's just hard for static analysis tools to see it.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable pylint misplaced-comparison-constant warning
Brian Foley [Tue, 18 Oct 2016 19:20:12 +0000 (20:20 +0100)]
Disable pylint misplaced-comparison-constant warning

It's correct, and easier to read a < x < b in this case.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable incorect pylint simplify-if-statement warning
Brian Foley [Tue, 18 Oct 2016 19:01:04 +0000 (20:01 +0100)]
Disable incorect pylint simplify-if-statement warning

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable pylint eval-used warning
Brian Foley [Mon, 17 Oct 2016 15:00:03 +0000 (16:00 +0100)]
Disable pylint eval-used warning

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable pylint invalid-name warning
Brian Foley [Mon, 17 Oct 2016 14:07:56 +0000 (15:07 +0100)]
Disable pylint invalid-name warning

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable pylint import-self warning
Brian Foley [Mon, 17 Oct 2016 14:07:43 +0000 (15:07 +0100)]
Disable pylint import-self warning

In this case it's a deliberate test to make sure that the contents
of lib/ have been correctly installed in a dir called ganeti.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDisable some pylint unused-import warnings
Brian Foley [Mon, 17 Oct 2016 13:52:52 +0000 (14:52 +0100)]
Disable some pylint unused-import warnings

Technically the imports are unused in the module, but they're being
imported to make them available to users of the module.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoReplace deprecated pylint >=0.27 pragma with new form
Brian Foley [Sun, 4 Dec 2016 20:19:43 +0000 (20:19 +0000)]
Replace deprecated pylint >=0.27 pragma with new form

The disable-all pragma was deprecated in favour of the new skip-file
pragma in pylint >= 0.27. Since we're moving to 1.6.4, use this instead.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoDelete old warning disables removed from pylint 1.6
Brian Foley [Tue, 18 Oct 2016 20:37:33 +0000 (21:37 +0100)]
Delete old warning disables removed from pylint 1.6

These include star-args and r0924: badly implemented immutable.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoFix pylint >1.4 pycurl no-member warnings
Brian Foley [Sun, 4 Dec 2016 18:49:16 +0000 (18:49 +0000)]
Fix pylint >1.4 pycurl no-member warnings

Add pycurl to the whitelist of C extensions that can be loaded for the
purposes of attribute checking. This works around a security feature
introduced in pylint 1.4 to avoid arbitrary code injection.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: Remove unused/duplicate module/fn import
Brian Foley [Tue, 18 Oct 2016 18:33:39 +0000 (19:33 +0100)]
Cleanup: Remove unused/duplicate module/fn import

No functional change. Found by pylint's reimported warning.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: Fix unidiomatic-typecheck
Brian Foley [Tue, 18 Oct 2016 17:29:15 +0000 (18:29 +0100)]
Cleanup: Fix unidiomatic-typecheck

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: Remove some unneeded pylint disables
Brian Foley [Tue, 18 Oct 2016 17:22:52 +0000 (18:22 +0100)]
Cleanup: Remove some unneeded pylint disables

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: Iterate dict rather than key list
Brian Foley [Mon, 17 Oct 2016 16:49:50 +0000 (17:49 +0100)]
Cleanup: Iterate dict rather than key list

The later explicitly constructs a new list of the dictionary's keys, which
we don't need here. Quietens pylint's consider-iterating-dictionary warning.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: Remove unused format key
Brian Foley [Mon, 17 Oct 2016 15:48:03 +0000 (16:48 +0100)]
Cleanup: Remove unused format key

No functional change, but quietens pylint unused-format-string-argument
warning.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: StartInstance and RebootInstance return None
Brian Foley [Mon, 17 Oct 2016 15:22:14 +0000 (16:22 +0100)]
Cleanup: StartInstance and RebootInstance return None

Quell assignment-from-none warning

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: Fix for/else with no break in AddAuthorizedKeys
Brian Foley [Sun, 4 Dec 2016 19:35:09 +0000 (19:35 +0000)]
Cleanup: Fix for/else with no break in AddAuthorizedKeys

After a refactoring in c178396, the only break statement in the for/else
loop in AddAuthorizedKeys was removed. This means the else block is
always executed, so remove the else: to quell a pylint warning.

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: Replace map/filters with list comprehensions
Brian Foley [Mon, 17 Oct 2016 14:54:27 +0000 (15:54 +0100)]
Cleanup: Replace map/filters with list comprehensions

No functional change, but more pythonic and quells pylint's
deprecated-lambda warning.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: del is a statement not a function
Brian Foley [Mon, 17 Oct 2016 13:16:40 +0000 (14:16 +0100)]
Cleanup: del is a statement not a function

Remove unnecessary parens to make pylint happy.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: Use FOO not in BAR instead of not FOO in BAR
Brian Foley [Mon, 17 Oct 2016 13:14:31 +0000 (14:14 +0100)]
Cleanup: Use FOO not in BAR instead of not FOO in BAR

No functional change, but more pythonic, and makes pylint's
unneeded-not warning happy.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: Simplify boolean assignment
Brian Foley [Fri, 2 Dec 2016 17:14:32 +0000 (17:14 +0000)]
Cleanup: Simplify boolean assignment

No functional change. Found by pylint's simplifiable-if-statement.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoCleanup: Remove some unnecessary if (...) parens
Brian Foley [Fri, 2 Dec 2016 17:11:02 +0000 (17:11 +0000)]
Cleanup: Remove some unnecessary if (...) parens

No functional change, but makes pylint 1.1.0 a little happier, and
is more pythonic.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoAdd OS params removal in gnt-instance reinstall
Yiannis Tsiouris [Mon, 5 Dec 2016 14:41:39 +0000 (16:41 +0200)]
Add OS params removal in gnt-instance reinstall

This patch adds '--remove-os-parameters' and
'--remove-os-parameters-private' options to 'gnt-instance reinstall'.
Similarly to 'gnt-instance modify', the new options can be used to
perform reinstalls to OS providers that support different parameters.
E.g.:

$ gnt-instance reinstall --remove-os-parameters param1,param2 \
  -o <new_os_provider> -O param3=val3 <instance_name>

or

$ gnt-instance reinstall --remove-os-parameters param1,param2 \
  -o <new_os_provider> -O param3=val3 <instance_name>

Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoSupport clear OS params in gnt-instance reinstall
Yiannis Tsiouris [Mon, 5 Dec 2016 14:41:38 +0000 (16:41 +0200)]
Support clear OS params in gnt-instance reinstall

This extends 'gnt-instance reinstall' to support
'--clear-os-parameters' and '--clear-os-parameters-private' options from
'gnt-instance modify'. E.g.:

$ gnt-instance reinstall --clear-os-parameters -o <new_os_provider> \
  -O param1=val1,param2=val2 <instance_name>

or

$ gnt-instance reinstall --clear-os-parameters-private \
  -o <new_os_provider> -O param1=val1,param2=val2 <instance_name>

Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoTest new options of gnt-instance modify in RAPI
Yiannis Tsiouris [Mon, 5 Dec 2016 14:41:37 +0000 (16:41 +0200)]
Test new options of gnt-instance modify in RAPI

This patch adds tests to RAPI for gnt-instance modify options:
- '--clear-os-parameters'/'--clear-os-parameters-private'
- '--remove-os-parameters'/'--remove-os-parameters-private'

Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoAdd cmdlib tests for removing instance OS params
Nikos Skalkotos [Mon, 5 Dec 2016 14:41:36 +0000 (16:41 +0200)]
Add cmdlib tests for removing instance OS params

Add LUInstanceSetParams tests for removing individual public & private
parameters, as well as, for clearing out public & private parameters.

Signed-off-by: Nikos Skalkotos <skalkoto@grnet.gr>
Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoSupport OS params removal in gnt-instance modify
Yiannis Tsiouris [Mon, 5 Dec 2016 14:41:35 +0000 (16:41 +0200)]
Support OS params removal in gnt-instance modify

This patch extends 'gnt-instance modify' by allowing a user to remove a
list of public/private OS parameters from an instance. This can be
useful before performing a reinstall to a new OS provider. Example
usage:

$ gnt-instance modify --remove-os-parameters parm1,parm2 <instance_name>

or

$ gnt-instance modify --remove-os-parameters-private parm3 <instance_name>

Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoAdd clear OS params options in gnt-instance modify
Yiannis Tsiouris [Mon, 5 Dec 2016 14:41:34 +0000 (16:41 +0200)]
Add clear OS params options in gnt-instance modify

This patch adds two new options in 'gnt-instance modify' which allow the
user to clear all current public/private OS parameters of an instance.
This might be useful for OS providers that consider no parameters as
valid or, more commonly, for changing OS provider (and parameters)
before performing a reinstall. E.g.:

$ gnt-instance modify --clear-os-parameters <instance_name>

or

$ gnt-instance modify --clear-os-parameters-private <instance_name>

Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoFix invalid variable error for file-based disks
Brian Foley [Mon, 17 Oct 2016 16:27:26 +0000 (17:27 +0100)]
Fix invalid variable error for file-based disks

Introduced by 94e252a33. Found by pylint's undefined-loop-variable
warning.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

2 years agoFIX: Refactor DiagnoseOS to use a loop, not an inner fn
Brian Foley [Mon, 17 Oct 2016 13:24:26 +0000 (14:24 +0100)]
FIX: Refactor DiagnoseOS to use a loop, not an inner fn

Found by pylint's cell-var-from-loop.

The inner function tries to access a dictionary that's local to the
outer function, assuming that Python has similar lexical scoping rules
to most languages. Unfortunately it doesn't. See
https://docs.python.org/3/tutorial/classes.html#python-scopes-and-namespaces

Refactor code to work as intended by moving the fn body into a loop.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

2 years agoFIX: Set INSTANCE_NICn_NETWORK_NAME only if net is defined
Brian Foley [Mon, 17 Oct 2016 13:08:32 +0000 (14:08 +0100)]
FIX: Set INSTANCE_NICn_NETWORK_NAME only if net is defined

Found by pylint's using-constant-test warning. 'network' is a module,
'net' is the network name for this NIC.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

2 years agoStartInstance restores instance state if running
Viktor Bachraty [Fri, 7 Oct 2016 19:10:34 +0000 (20:10 +0100)]
StartInstance restores instance state if running

Add a side effect to StartInstance on Xen hypervisor that restores the
Xen configuration in case it is missing. This is a common failure
scenario during migrations. The generic backend checks if the instance
is running and queries the hypervisor about it's state. If it needs
recovery, the backend gathers the associated block devices and calls
RestoreInstance in the same manner as if it would call StartInstance in
case the instance wasn't already running.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoAllow migrate --cleanup to adopt an instance
Viktor Bachraty [Fri, 7 Oct 2016 17:57:22 +0000 (18:57 +0100)]
Allow migrate --cleanup to adopt an instance

In some failure modes, Ganeti state of record may desync from actual
state of world. This patch allows migrate --cleanup to adopt an instance
if it is detected running on an unexpected node.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoMake migrate --cleanup more robust
Viktor Bachraty [Fri, 7 Oct 2016 16:29:23 +0000 (17:29 +0100)]
Make migrate --cleanup more robust

In case of Xen migrations, the most common failure case is when the
instance fails to freeze so the migration fails with domains running on
both target and source node. This patchs allows migrate --cleanup to
recover by running AbortMigrate() in case the instance is running on
both the source and target node.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoMake finalize_migration_{src,dst} a single op
Viktor Bachraty [Fri, 7 Oct 2016 16:17:44 +0000 (17:17 +0100)]
Make finalize_migration_{src,dst} a single op

FinalizeMigrationSource and FinalizeMigrationDst should compose an
single operation consisting of 2 idempotent steps, that can be always
both retried, similarly how they are used in AbortMigration() without
any reliance on the order of their execution. Otherwise we would have to
remember at which step did the migration fail in order to be able to
rollback/recover. Also state of the record should be updated only if
both succeeded, otherwise we won't know in case of recovery if the
record corresponds to the original state or the final state.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoMake FinalizeMigration{Src,Dst} more robust
Viktor Bachraty [Fri, 7 Oct 2016 16:12:23 +0000 (17:12 +0100)]
Make FinalizeMigration{Src,Dst} more robust

Migration is supported on KVM/Xen hypervisors only. Finalize methods are
called in both success or suspected failure (the master node can't
certainly know about failures - e.g. in case of timeouts). Thus these
functions need to be as robust as possible in order to be able to
recover from failures - in case success=False the functions should be
conservative, trying to detect the actual state and idempotent. The
cleanup workaround works for XM, on XL it is a no-op.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoFix instance state detection in _Shutdowninstance
Viktor Bachraty [Thu, 6 Oct 2016 18:52:49 +0000 (19:52 +0100)]
Fix instance state detection in _Shutdowninstance

GetInstanceInfo returns info that is already parsed with state being an
enum value converted from string using _XenToHypervisorInstanceState()

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoCode cleanup in hypervisor backend
Viktor Bachraty [Thu, 6 Oct 2016 18:34:51 +0000 (19:34 +0100)]
Code cleanup in hypervisor backend

Non-functional changes, smaller code cleanup.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoFix for incorrect parsing of DRBD versions
Federico Morg Pareschi [Fri, 2 Dec 2016 11:12:25 +0000 (11:12 +0000)]
Fix for incorrect parsing of DRBD versions

Following issue #1194, this patch allows Ganeti to correctly
parse drbd versions that also include a dash in their k-fix
version component.

This technically makes Ganeti treat x.y.z.k the same as x.y.z-k
with regards to compatibility. For practicality we can assume that
these versions are the same, this is the intended behaviour as it is
unlikely that these two versions would be incompatible with each
other, if that ever were to happen.

Signed-off-by: Federico Morg Pareschi <morg@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoFix for instance reinstall not updating config
Federico Morg Pareschi [Mon, 28 Nov 2016 13:58:06 +0000 (13:58 +0000)]
Fix for instance reinstall not updating config

As per issue #1193, instance reinstalls with modified OS parameters
did not persist among reinstalls and were not saved to the cluster's
config data. With this patch, this problem should be fixed.

Signed-off-by: Federico Morg Pareschi <morg@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

2 years agoChange a few errors to report names, not UUIDs
Brian Foley [Thu, 29 Sep 2016 13:04:35 +0000 (14:04 +0100)]
Change a few errors to report names, not UUIDs

Node/Node group name is much easier for humans to read.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

2 years agoGive atomicWriteFile temp filenames a more distinct pattern
Brian Foley [Wed, 28 Sep 2016 13:25:49 +0000 (14:25 +0100)]
Give atomicWriteFile temp filenames a more distinct pattern

This stops them being confused with 'real' filenames by both users and
code if the daemon crashes before the temporary file is removed.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

2 years agoFirst implementation of the predictive scheduler
Federico Morg Pareschi [Thu, 15 Sep 2016 12:55:33 +0000 (13:55 +0100)]
First implementation of the predictive scheduler

This is just a partial implementation of the static lock prediction system.
With this, Ganeti queue system tries to predict the locks of the running
jobs and schedule new jobs that are less likely to block.
It is a partial implementation which might cause starvation for jobs stuck
in the queue, following patches will address this issue.

Signed-off-by: Federico Morg Pareschi <morg@google.com>
Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

2 years agoLV check failure should print instance name
Viktor Bachraty [Wed, 14 Sep 2016 12:08:17 +0000 (13:08 +0100)]
LV check failure should print instance name

Instances with secondaries in a separate nodegroup from primary can
result in cluster verify throwing a missing node lock error for LV
check. This should properly print the node name and the violating
instance name.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

2 years agoFix RAPI auth bug introduced by f3b51166
Brian Foley [Sun, 11 Sep 2016 21:20:48 +0000 (21:20 +0000)]
Fix RAPI auth bug introduced by f3b51166

This accidentally dropped the RemoteApiHandler.AuthenticationRequired
logic to check if a specific RAPI call handler required auth for
read/write access to the cluster config.

This wasn't noticed because some unit tests were changed to hardcode
auth to be required unconditionaly.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

2 years agoAdd ganeti-noded and ganeti-rapi --max-clients options
Brian Foley [Mon, 12 Sep 2016 14:52:49 +0000 (15:52 +0100)]
Add ganeti-noded and ganeti-rapi --max-clients options

This defaults to 20 -- the same as the current hardcoded limit.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

2 years agoDisable logging CallRPCMethod timings in non-debug configs
Brian Foley [Mon, 12 Sep 2016 13:24:46 +0000 (14:24 +0100)]
Disable logging CallRPCMethod timings in non-debug configs

It's been useful information when debugging performance issues in
2.15/2.16, but not useful enough to leave turned on in non-debug builds
as it clutters up commands.log, jobs.log, rapi-daemon.log, watcher.log

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

2 years ago568 Update hv_kvm to handle output from qemu >= 1.6.0
Brian Foley [Mon, 5 Sep 2016 15:38:53 +0000 (16:38 +0100)]
568 Update hv_kvm to handle output from qemu >= 1.6.0

In qemu 1.6 and later, qemu info migrate changed from
    ...
    transferred ram: 980546 kbytes
    remaining ram: 0 kbytes
    total ram: 1049024 kbytes
    ...
to
    transferred ram: 61460 kbytes
    throughput: 671.42 mbps
    remaining ram: 358212 kbytes
    total ram: 1049672 kbytes
    ...

This stops migrates from showing progress for memory transfers that
take a long time. Update the regexp to handle this (should work with
up to at least qemu 2.7)

Thanks to danielk_lists@z9d.de for the fix.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

2 years agoAdd design document for a predictive queue system
Federico Morg Pareschi [Tue, 30 Aug 2016 15:11:24 +0000 (16:11 +0100)]
Add design document for a predictive queue system

This design document introduces a new Ganeti job scheduler that
prioritizes jobs in the queue based on their lock declaration and state
of the cluster.

Signed-off-by: Federico Morg Pareschi <morg@google.com>
Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

3 years ago1143: Add gnt-cluster modify --modify-ssh-setup option
Brian Foley [Wed, 17 Aug 2016 13:56:17 +0000 (14:56 +0100)]
1143: Add gnt-cluster modify --modify-ssh-setup option

This allows the cluster flag modify_ssh_setup to be changed after
cluster initialisation.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

3 years agoImprove cluster verify ssh key errors
Brian Foley [Wed, 17 Aug 2016 17:34:38 +0000 (18:34 +0100)]
Improve cluster verify ssh key errors

Make it clearer which keys in which locations we're referring to.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

3 years agoFix hlint warnings introduced by 02914f7d
Brian Foley [Mon, 22 Aug 2016 11:41:38 +0000 (12:41 +0100)]
Fix hlint warnings introduced by 02914f7d

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

3 years agoMerge branch 'stable-2.17'
Brian Foley [Mon, 22 Aug 2016 11:23:36 +0000 (12:23 +0100)]
Merge branch 'stable-2.17'

* stable-2.17
  Fix hlint warnings

* stable-2.16
  Fix inconsistent spaces vs tabs indent in makefile
  Fix some typos/poor phrasing in gnt-node man page
  Add missing cluster modify --modify-etc-hosts to man
  Fix/quell hlint warnings

* stable-2.13
  Bugfix: migrate needs HypervisorClass, not an instance

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

3 years agoMerge branch 'stable-2.16' into stable-2.17
Brian Foley [Mon, 22 Aug 2016 11:17:21 +0000 (12:17 +0100)]
Merge branch 'stable-2.16' into stable-2.17

* stable-2.16
  Fix inconsistent spaces vs tabs indent in makefile

* stable-2.13
  Bugfix: migrate needs HypervisorClass, not an instance

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

3 years agoMerge branch 'stable-2.15' into stable-2.16
Brian Foley [Mon, 22 Aug 2016 11:08:27 +0000 (12:08 +0100)]
Merge branch 'stable-2.15' into stable-2.16

* stable-2.13
  Bugfix: migrate needs HypervisorClass, not an instance

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

3 years agoMerge branch 'stable-2.14' into stable-2.15
Brian Foley [Mon, 22 Aug 2016 10:59:42 +0000 (11:59 +0100)]
Merge branch 'stable-2.14' into stable-2.15

* stable-2.13
  Bugfix: migrate needs HypervisorClass, not an instance

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

3 years agoMerge branch 'stable-2.13' into stable-2.14
Brian Foley [Mon, 22 Aug 2016 10:40:10 +0000 (11:40 +0100)]
Merge branch 'stable-2.13' into stable-2.14

* stable-2.13
  Bugfix: migrate needs HypervisorClass, not an instance

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

3 years agoBugfix: migrate needs HypervisorClass, not an instance stable-2.13
David Mohr [Mon, 22 Aug 2016 10:31:47 +0000 (11:31 +0100)]
Bugfix: migrate needs HypervisorClass, not an instance

Otherwise it will complain about permissions of
/var/run/ganeti/kvm-hypervisor

Signed-off-by: David Mohr <david@mcbf.net>
Reviewed-by: Brian Foley <bpfoley@google.com>