ganeti-github.git
2 years agoMerge branch 'stable-2.16' into 'upmerge-2.17' branch upmerge-2.17
Rafael Marinheiro [Thu, 9 Nov 2017 16:02:04 +0000 (16:02 +0000)]
Merge branch 'stable-2.16' into 'upmerge-2.17' branch

This PR merges the changes from the stable-2.16 branch into the stable-2.17 branch.

Reviewed-on: #1271
Signed-off-by: Federico "Morg" Pareschi <morg@google.com>
Reviewed-by: Rafael Marinheiro <marinheiro@google.com>

2 years agoMerge branch 'stable-2.16' into upmerge-2.17
Federico Morg Pareschi [Tue, 24 Oct 2017 11:00:46 +0000 (12:00 +0100)]
Merge branch 'stable-2.16' into upmerge-2.17

* stable-2.16
  hv_xen: generate correct type for paravirtualized nic (#57)
  Force CleanupInstance always on InstanceShutdown (#53)
  Cleanup blockdevs from target on migration failure (#50)
  kvm: Add missing 'driver' in 'hvinfo' dict (#46)
  Add cleanup of stale OS hvp data on cluster modify
  Fix: changed params in setInstanceLocationScore
  Consider state-of-the-record free memory in htools memory model (#35)
  Add optStaticKvmNodeMemory to HTools and IAllocator (#34)
  Htools should use state-of-record instance size
  Force "sudo: false" on Travis-CI config file
  Adding travis-ci support
  Ceph/RBD rbd showmapped -p is no longer supported
  Adding .gitignore entry for the maintd binary
  Fix missing import in RAPI server
  Add discard KVM option to bdev
  kvm: use_guest_agent: QEMU Guest Agent support

* stable-2.15
  Add Ganeti CONTRIBUTE.md
  Add test for removing disk on a live instance
  Prohibit disk removal w/o hotplug on live instance
  Fix tuple-unpacking from QueryInstances result
  Fix idx in RemoveDisks warning
  Make man pages more consistent in parameter format
  Re-add 2 imports incorrectly removed during cleanup
  Add status and "version implemented" fields to design docs
  Fix typo in cli_opts.py IGNORE_HVVERSIONS_OPT flag
  Fix coexistence of location tags and non-DRBD instances
  Fix backup export in case of ext disk template
  Fix miscellaneous typos found when reading design docs
  Revert "Merge branch 'stable-2.15' into stable-2.16"i

Merge-related fixes:

  Fixed a conflict between old 2.16 and new 2.17 memory model by porting
  the 2.16 changes into the 2.17 branch and make them work with the memory
  overcommittment changes.

Signed-off-by: Federico Morg Pareschi <morg@google.com>

2 years agohv_xen: generate correct type for paravirtualized nic (#57)
Richard Kojedzinszky [Tue, 11 Jul 2017 13:07:15 +0000 (15:07 +0200)]
hv_xen: generate correct type for paravirtualized nic (#57)

With debian stretch (ganeti 2.15 and xen 4.8) the configuration file
generated by ganeti is not correct, type=vif should be specified instead of
type=paravirtualized when using paravirtual nics.

Reviewed-by: Federico Morg Pareschi <morg@google.com>

2 years agoForce CleanupInstance always on InstanceShutdown (#53)
Yiannis Tsiouris [Tue, 11 Jul 2017 10:50:17 +0000 (13:50 +0300)]
Force CleanupInstance always on InstanceShutdown (#53)

This commit fixes the bug described in #52 by always trying to cleanup
an instance's runtime files and block devices on InstanceShutdown. The
buggy scenario is:
- A user issues a 'poweroff' from inside the VM
- The admin or a RAPI client does a 'gnt-instance shutdown' (or even
worse a 'gnt-instance remove' which, in turn, issues an instance
shutdown) *before* the watcher notices the 'ERROR_down' state of the
VM
- The instance status changes to 'ADMIN_down' (or gets removed) but the
cleanup is never performed. Thus, the CTRL_DIR, CONF_DIR, NIC_DIR,
instance_disks, etc. are left stale.

This can be reproduced on clusters that have the 'user_shutdown' option
disabled.

Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Signed-off-by: Alexandros Kiousis <alexk@noc.grnet.gr>
Reviewed-by: Federico Morg Pareschi <morg@google.com>

2 years agoCleanup blockdevs from target on migration failure (#50)
Yiannis Tsiouris [Mon, 10 Jul 2017 10:33:08 +0000 (13:33 +0300)]
Cleanup blockdevs from target on migration failure (#50)

When an instance migration fails (e.g. hypervisor failure or
'call_accept_instance' failure), stale block devices remain on
target node. This commit fixes this by adding cleanup code in
'_RevertDiskStatus' for RBD and EXT types.

Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Reviewed-by: Federico Morg Pareschi <morg@google.com>

3 years agokvm: Add missing 'driver' in 'hvinfo' dict (#46)
Yiannis Tsiouris [Fri, 2 Jun 2017 10:42:48 +0000 (13:42 +0300)]
kvm: Add missing 'driver' in 'hvinfo' dict (#46)

This commit fixes a bug introduced in 28639115d7 where the 'driver'
option was missing from the 'hvinfo' dict when
'_UpgradeSerializedRuntime' was called. This raised a KeyError when
trying to migrate a running instance.

Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Signed-off-by: Alexandros Afentoulis <alexaf@noc.grnet.gr>
Reviewed-by: Federico Morg Pareschi <morg@google.com>

3 years agoAdd Ganeti CONTRIBUTE.md
Rafael Marinheiro [Fri, 7 Apr 2017 16:11:20 +0000 (17:11 +0100)]
Add Ganeti CONTRIBUTE.md

Add CONTRIBUTE.md, which details the contribution process for Ganeti.

Reviewed-on: https://github.com/ganeti/ganeti/pull/40
Signed-off-by: Rafael Marinheiro <marinheiro@google.com>
Reviewed-by: Federico "Morg" Pareschi <morg@google.com>

3 years agoAdd Ganeti CONTRIBUTE.md 40/head
Rafael Marinheiro [Fri, 7 Apr 2017 14:07:01 +0000 (15:07 +0100)]
Add Ganeti CONTRIBUTE.md

Add CONTRIBUTE.md, which details the contribution process for Ganeti.

Reviewed-on: https://github.com/ganeti/ganeti/pull/39
Signed-off-by: Rafael Marinheiro <marinheiro@google.com>
Reviewed-by: Federico "Morg" Pareschi <morg@google.com>

3 years agoAdd cleanup of stale OS hvp data on cluster modify
Morg [Tue, 4 Apr 2017 14:43:58 +0000 (15:43 +0100)]
Add cleanup of stale OS hvp data on cluster modify

Cluster config would keep track of old/stale OS data even if their
hypervisor parameters are already gone. This check makes sure to wipe
them from the config dictionary if they are empty.

Reviewed-on: https://github.com/ganeti/ganeti/pull/38
Signed-off-by: Federico Morg Pareschi <morg@google.com>
Reviewed-by: Rafael Marinheiro <marinheiro@google.com>

3 years agoMerge branch 'upmerge-2.16' into stable-2.16
Rafael Marinheiro [Thu, 30 Mar 2017 14:43:54 +0000 (15:43 +0100)]
Merge branch 'upmerge-2.16' into stable-2.16

* stable-2.15
  Add test for removing disk on a live instance
  Prohibit disk removal w/o hotplug on live instance
  Fix tuple-unpacking from QueryInstances result
  Fix idx in RemoveDisks warning
  Make man pages more consistent in parameter format
  Re-add 2 imports incorrectly removed during cleanup
  Add status and "version implemented" fields to design docs
  Fix typo in cli_opts.py IGNORE_HVVERSIONS_OPT flag
  Fix coexistence of location tags and non-DRBD instances
  Fix backup export in case of ext disk template
  Fix miscellaneous typos found when reading design docs

Merge conflicts:

  src/Ganeti/HTools/Cluster.hs # removed block, kept 2.16 version
  man/gnt-instance.rst # slash in new_name
  man/gnt-cluster.rst # dash in cluster-name
  lib/client/gnt_cluster.py # angle brackets
  doc/design-shared-storage-redundancy.rst # removed blcok, kept 2.16 version
  doc/design-draft.rst # kept 2.16 specific list

Merge-related fixes:
  Fixes a build error caused by the change of the interface of
    setInstanceLocationScore

Reviewed-on: https://github.com/ganeti/ganeti/pull/37
Signed-off-by: Rafael Marinheiro <marinheiro@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

3 years agoMerge stable-2.15 into stable-2.16 upmerge-2.16 37/head
Rafael Marinheiro [Wed, 29 Mar 2017 16:13:23 +0000 (17:13 +0100)]
Merge stable-2.15 into stable-2.16

Merge branch 'stable-2.15' into stable-2.16

Reviewed-on: https://github.com/ganeti/ganeti/pull/36
Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Rafael Marinheiro <marinheiro@google.com>

3 years agoFix: changed params in setInstanceLocationScore 36/head
Viktor Bachraty [Wed, 29 Mar 2017 09:29:15 +0000 (10:29 +0100)]
Fix: changed params in setInstanceLocationScore

Patch 8fcb5ec531 fixing a bug in 2.15 changed the interface of
setInstanceLocationScore.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>

3 years agoMerge branch 'stable-2.15' into stable-2.16
Viktor Bachraty [Tue, 28 Mar 2017 16:49:23 +0000 (17:49 +0100)]
Merge branch 'stable-2.15' into stable-2.16

* stable-2.15
  Add test for removing disk on a live instance
  Prohibit disk removal w/o hotplug on live instance
  Fix tuple-unpacking from QueryInstances result
  Fix idx in RemoveDisks warning
  Make man pages more consistent in parameter format
  Re-add 2 imports incorrectly removed during cleanup
  Add status and "version implemented" fields to design docs
  Fix typo in cli_opts.py IGNORE_HVVERSIONS_OPT flag
  Fix coexistence of location tags and non-DRBD instances
  Fix backup export in case of ext disk template
  Fix miscellaneous typos found when reading design docs

Merge conflicts:

  src/Ganeti/HTools/Cluster.hs # removed block, kept 2.16 version
  man/gnt-instance.rst # slash in new_name
  man/gnt-cluster.rst # dash in cluster-name
  lib/client/gnt_cluster.py # angle brackets
  doc/design-shared-storage-redundancy.rst # removed blcok, kept 2.16 version
  doc/design-draft.rst # kept 2.16 specific list

Signed-off-by: Viktor Bachraty <vbachraty@google.com>

3 years agoMerge commit '8048e563' into stable-2.16
Viktor Bachraty [Tue, 28 Mar 2017 16:28:51 +0000 (17:28 +0100)]
Merge commit '8048e563' into stable-2.16

We use -s ours to merge this particular patch, because it is a revert
of a mistake that we want to keep in 2.15 only.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>

3 years agoConsider state-of-the-record free memory in htools memory model (#35)
Viktor Bachraty [Tue, 28 Mar 2017 14:45:20 +0000 (15:45 +0100)]
Consider state-of-the-record free memory in htools memory model (#35)

The existing memory model in htools was reporting incorrect node memory,
missing memory and also available (free) memory on KVM that resulted in some
cases in overcramming nodes (overcommiting memory without having swap to evict
pages from RAM) leading to OOM kills.

This model is equivalent to the model proposed in issue 127, but doesn't
introduce overcommitment (swap or balooning) support nor does it depend
on Ganeti 2.17 features.

The goal is to distinguish between reported (state-of-world) values and recorded
(state-of-record) values and not to mix them arbitrarily. To estimate free
memory we always need to take the minimum of measured/calculated values using
those two states (root causes of the differences explained in missing memory) in
order to be on the safe side.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Rafael Marinheiro <marinheiro@google.com>

3 years agoAdd optStaticKvmNodeMemory to HTools and IAllocator (#34)
Viktor Bachraty [Tue, 28 Mar 2017 11:07:58 +0000 (12:07 +0100)]
Add optStaticKvmNodeMemory to HTools and IAllocator (#34)

This is an attempt to fix balancing on 2.16 using KVM hypervisor. The problem
with KVM is, that it reports 'active' memory from /proc/memstat for node
memory a.k.a. nMem a.k.a. memory_dom0.

This is plain wrong, as 'active' memory in Linux includes memory from all
running processes, including instances (qemu processes). We can't calculate
the node memory on the hypervisor side, simply subtracting RES used by qemu
processes would give us only approximate value and still it would be the actively
used memory rather than a upper hard limit as in dom0 size in Xen. As KVM
does not provide proper isolation for node memory, the only thing we can do is
to instruct htools to reserve a static value for node memory.

This is solved in 2.17 in patch f971341471 implementing the obtainNodeMemory
function that takes the live dom0 size in Xen, but uses 'hv_state' for exporting
the node size configurable parameter for KVM. I've opted for _not_ backporting
the cluster parameter from 2.17 for these reasons:

* this way the change is isolated to hbal, no need to change cluster config
* the obtainNodeMemory is for Luxi/Rapi only, this flag works with any backend
* the goal is to fix balancing as it is, without adding support for memory
overcommitment, doing so would require backporting a bunch of other dependencies:
d2bfc50608f971341471 and possibly more.

The flag is ignored on Xen, and on KVM is set by default to 4G, so the change
should be backward compatible (no need to specify this flag unless you need
a different value).

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Rafael Marinheiro <marinheiro@google.com>

3 years agoHtools should use state-of-record instance size
Viktor Bachraty [Mon, 27 Mar 2017 18:10:45 +0000 (19:10 +0100)]
Htools should use state-of-record instance size

The actual memory usage is irrelevant, because even if the instance uses less
memory, it can decide anytime to touch the yet unused portion, so we can't take
that away from the instance without risking OOM conditions (on KVM). If it uses
more, it will be accounted as missing memory.

In case of oversubscription (unsupported yet) we have the following mechanisms:

KVM: Memory manager has to evict some pages to swap. That means:
 swap + physical_memory > sum(all instances state-of-record size) + node_memory

Xen: Balooning has to ask the instance's permission to take some pages by
allocating a buffer in the instance's balooning kernel driver. Thus the instance
isn't in fact resized, just some pages are borrowed and reused in the free pool.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Rafael Marinheiro <marinheiro@google.com>

3 years agoForce "sudo: false" on Travis-CI config file
Rafael Marinheiro [Wed, 22 Mar 2017 19:04:21 +0000 (19:04 +0000)]
Force "sudo: false" on Travis-CI config file

We should use the container build to properly isolate our dependencies. In order
to do this we have to run the build using "sudo: false". The current
configuration file works fine for recent forks because the default for them is
"sudo: false", while the default for older repos is "sudo: required" (see [1]).

[1]: https://docs.travis-ci.com/user/ci-environment/#Virtualization-environments

Signed-off-by: Rafael Marinheiro <marinheiro@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

3 years agoAdding travis-ci support
Rafael Marinheiro [Wed, 22 Mar 2017 16:18:48 +0000 (16:18 +0000)]
Adding travis-ci support

Travis-ci configuration yml.

Signed-off-by: Rafael Marinheiro <marinheiro@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

3 years agoCeph/RBD rbd showmapped -p is no longer supported
Ansgar Jazdzewski [Wed, 8 Mar 2017 15:36:17 +0000 (15:36 +0000)]
Ceph/RBD rbd showmapped -p is no longer supported

In order to solve the missing "-p" option in "rbd showmapped" it is
removed from the showmap_cmd. To make sure we found the volume in the
showmapped (json) output, we now check also the pool in the
_ParseRbdShowmappedJson method.

Signed-off-by: Ansgar Jazdzewski <a.jazdzewski@googlemail.com>
Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Federico Morg Pareschi <morg@google.com>

3 years agoAdding .gitignore entry for the maintd binary
Rafael Marinheiro [Tue, 7 Mar 2017 17:15:09 +0000 (17:15 +0000)]
Adding .gitignore entry for the maintd binary

Signed-off-by: Rafael Marinheiro <marinheiro@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

3 years agoFix missing import in RAPI server
Yiannis Tsiouris [Mon, 20 Feb 2017 15:01:46 +0000 (17:01 +0200)]
Fix missing import in RAPI server

This commit fixes a bug introduced in 7b1fd2e60 after pylint cleanup of
"unused" imports. We also need to suppress this warning as pylint cannot
determine that this import was actually used by the
'http.server.HttpServerHandler' symbol.

Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

3 years agoAdd discard KVM option to bdev
Viktor Bachraty [Mon, 6 Feb 2017 20:10:11 +0000 (20:10 +0000)]
Add discard KVM option to bdev

It can be helpful if you are using ext-storage (iSCSI) or ceph
(RBD) so free space on a SAN is possible. For backward compatibility
the 'default' value will not pass any option to qemu.

Signed-off-by: Ansgar Jazdzewski <a.jazdzewski@googlemail.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

3 years agoAdd test for removing disk on a live instance
Yiannis Tsiouris [Thu, 26 Jan 2017 15:14:12 +0000 (17:14 +0200)]
Add test for removing disk on a live instance

Add a unit test for removing a disk from a running instance without
using hotplug.

Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Signed-off-by: Filippos Giannakos <philipgian@grnet.gr>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

3 years agoProhibit disk removal w/o hotplug on live instance
Yiannis Tsiouris [Thu, 26 Jan 2017 15:14:11 +0000 (17:14 +0200)]
Prohibit disk removal w/o hotplug on live instance

Prior to this commit, removing a disk from a live instance without
using hotplug succeeded incorrectly leaving stale disks behind. This is
due to the fact that the hypervisor holds the device open while Ganeti
tries to shut it down. The operation fails but Ganeti continues anyway,
removing the disk from the config and leaving the disk stale.

This patch adds an appropriate check before removing a disk from a live
instance.

Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Signed-off-by: Filippos Giannakos <philipgian@grnet.gr>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

3 years agoFix tuple-unpacking from QueryInstances result
Yiannis Tsiouris [Mon, 30 Jan 2017 13:18:33 +0000 (15:18 +0200)]
Fix tuple-unpacking from QueryInstances result

Prior to this commit, some code segments tried to unpack the return
value from 'QueryInstances' assuming that it's a single item list.
However, in some cases, e.g. 'gnt-instance console', the list might
have more results due to globbing. The patch handles this by always
selecting the first item of the return list.

Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Reviewed-by: Federico Morg Pareschi <morg@google.com>

3 years agoFix idx in RemoveDisks warning
Yiannis Tsiouris [Thu, 26 Jan 2017 21:03:27 +0000 (23:03 +0200)]
Fix idx in RemoveDisks warning

Before this patch, the warning of RemoveDisks was incorrectly showing
the index of the disk (whose removal failed) in the list of the
"pending-removal" disks instead of the, more general, instance's disk
list.

Signed-off-by: Yiannis Tsiouris <tsiou@grnet.gr>
Signed-off-by: Ivan Kanakarakis <ivan@grnet.gr>
Reviewed-by: Federico Morg Pareschi <morg@google.com>

3 years agoMake man pages more consistent in parameter format
Federico Morg Pareschi [Tue, 20 Dec 2016 15:10:27 +0000 (15:10 +0000)]
Make man pages more consistent in parameter format

This patch adds some more consistency in the parameter names
displayed in various ganeti man pages and their helper text
in gnt commands.

Signed-off-by: Federico Morg Pareschi <morg@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

3 years agoRe-add 2 imports incorrectly removed during cleanup
Brian Foley [Mon, 9 Jan 2017 11:29:11 +0000 (11:29 +0000)]
Re-add 2 imports incorrectly removed during cleanup

This was done in commit 7b1fd2e60e856ac to quell pylint (my bad!)

These broke ganeti-rapi and move-instance. Make sure, as a minimal
sanity check, that all the ganeti tools start up with --help.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agokvm: use_guest_agent: QEMU Guest Agent support
Robin H. Johnson [Fri, 6 Jan 2017 00:45:59 +0000 (16:45 -0800)]
kvm: use_guest_agent: QEMU Guest Agent support

Implement the QEMU Guest Agent sockets, so that code/scripts on the
hypervisors can communicate with guest operating systems easily.

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>
Reviewed-by: Brian Foley <bpfoley@google.com>

3 years agoAdd status and "version implemented" fields to design docs
Brian Foley [Wed, 4 Jan 2017 15:58:13 +0000 (15:58 +0000)]
Add status and "version implemented" fields to design docs

Based on a combination of reading NEWS, Ganeti version design docs, and
looking through the commit logs to see when features were implemented.

Clean up a number of inconsistencies and missed entries while here, and
update a number of (partly) implemented designs as no longer being
drafts.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

3 years agoFix typo in cli_opts.py IGNORE_HVVERSIONS_OPT flag
Federico Morg Pareschi [Tue, 3 Jan 2017 08:46:06 +0000 (08:46 +0000)]
Fix typo in cli_opts.py IGNORE_HVVERSIONS_OPT flag

Signed-off-by: Federico Morg Pareschi <morg@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

3 years agoFix coexistence of location tags and non-DRBD instances
Iustin Pop [Fri, 2 Dec 2016 22:03:55 +0000 (23:03 +0100)]
Fix coexistence of location tags and non-DRBD instances

This addresses issue 1185, “hbal: IntMap.!: key -1 is not an element of
the map”. The issue is that the location tags code presumed all
instances have a primary and a secondary (i.e., they are DRBD).

The fix is to set the location score for non-DRBD instances to zero,
since this is what I understand the correct value: such an instance
cannot be optimized by moving, so its score is "perfect" (zero).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

3 years agoFix backup export in case of ext disk template
Dimitris Aragiorgis [Wed, 4 Jan 2017 12:01:54 +0000 (14:01 +0200)]
Fix backup export in case of ext disk template

This commit fixes backup export in case of ext disk template
in two ways:

- Makes the generated snapshot (if any) inherit the params of the
  origin disk.

- Partially reverts commit 712ea2f that did export the origin disk
  instead of the snapshot one.

Fixes #1198.

Signed-off-by: Dimitris Aragiorgis <dimara@arrikto.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

3 years agoFix miscellaneous typos found when reading design docs
Brian Foley [Thu, 22 Dec 2016 15:26:19 +0000 (15:26 +0000)]
Fix miscellaneous typos found when reading design docs

Some of these are also in code comments, but don't touch any string
literals or error messages, so there should be no functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoMerge branch 'stable-2.16' into stable-2.17 stable-2.17
Brian Foley [Fri, 16 Dec 2016 17:23:46 +0000 (17:23 +0000)]
Merge branch 'stable-2.16' into stable-2.17

* stable-2.15
  Fix gnt-instance console instance unpausing for xl toolstack
  Disable pylint too-many-nested-blocks in _RunCmdPipe
  Reduce nesting in import-export ProcessChildIO
  Reduce nesting in LUOobCommand.Exec
  Reduce nesting in LUInstanceCreate.RunOsScripts
  Reduce nesting in RemoveNodeSshKeyBulk key calculation
  Reduce nesting in RemoveNodeSshKeyBulk ssh logic
  Reduce nesting in gnt-cluster VerifyDisks missing disk loop
  Reduce nesting in _CheckVLANArguments
  Reduce nesting in StartDaemon
  Disable pylint bad-continuation warning
  Disable pylint superfluous-parens warning
  Disable pylint redefined-variable-type warning
  Disable pylint too-many-branches warnings
  Disable pylint broad-except warnings
  Disable incorrect pylint assigning-non-slot warning
  Quell pylint unbalanced-tuple-unpacking warning
  Cleanup: Use new-style classes everywhere
  Quell pylint socket.timeout warning
  Quell the pylint wrong-import-order warnings
  Quell cell-var-from-loop warning
  Use default value lambda param to avoid cell-var-from-loop
  Quell too-many-boolean-expressions
  Remove pylint tests removed in pylint 2.0
  Quell trailing newline
  Quell bad-whitespace warning
  Quell consider-using-enumerate warning
  Disable pylint unsubscriptable-object warning
  Disable pylint bare-except warning
  Disable unwanted pylint wrong-import-position warnings
  Disable pylint unused-wildcard-import warning
  Disable incorrect pylint not-callable warning
  Disable pylint unpacking-non-sequence warning
  Disable pylint misplaced-comparison-constant warning
  Disable incorect pylint simplify-if-statement warning
  Disable pylint eval-used warning
  Disable pylint invalid-name warning
  Disable pylint import-self warning
  Disable some pylint unused-import warnings
  Replace deprecated pylint >=0.27 pragma with new form
  Delete old warning disables removed from pylint 1.6
  Fix pylint >1.4 pycurl no-member warnings
  Cleanup: Remove unused/duplicate module/fn import
  Cleanup: Fix unidiomatic-typecheck
  Cleanup: Remove some unneeded pylint disables
  Cleanup: Iterate dict rather than key list
  Cleanup: Remove unused format key
  Cleanup: StartInstance and RebootInstance return None
  Cleanup: Fix for/else with no break in AddAuthorizedKeys
  Cleanup: Replace map/filters with list comprehensions
  Cleanup: del is a statement not a function
  Cleanup: Use FOO not in BAR instead of not FOO in BAR
  Cleanup: Simplify boolean assignment
  Cleanup: Remove some unnecessary if (...) parens
  Fix invalid variable error for file-based disks
  FIX: Refactor DiagnoseOS to use a loop, not an inner fn
  FIX: Set INSTANCE_NICn_NETWORK_NAME only if net is defined
  StartInstance restores instance state if running
  Allow migrate --cleanup to adopt an instance
  Make migrate --cleanup more robust
  Make finalize_migration_{src,dst} a single op
  Make FinalizeMigration{Src,Dst} more robust
  Fix instance state detection in _Shutdowninstance
  Code cleanup in hypervisor backend
  Fix for incorrect parsing of DRBD versions
  Fix for instance reinstall not updating config
  Change a few errors to report names, not UUIDs
  Give atomicWriteFile temp filenames a more distinct pattern
  LV check failure should print instance name
  Add ganeti-noded and ganeti-rapi --max-clients options
  Disable logging CallRPCMethod timings in non-debug configs
  568 Update hv_kvm to handle output from qemu >= 1.6.0
  Improve cluster verify ssh key errors
  Fix inconsistent spaces vs tabs indent in makefile

* stable-2.13
  Bugfix: migrate needs HypervisorClass, not an instance

Fix easy merge conflict in lib/backend.py -- dead code removed in 2.15

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoMerge branch 'stable-2.15' into stable-2.16
Brian Foley [Fri, 16 Dec 2016 17:12:37 +0000 (17:12 +0000)]
Merge branch 'stable-2.15' into stable-2.16

Merge forward mostly cleanup patches from stable-2.15

* stable-2.15
  Fix gnt-instance console instance unpausing for xl toolstack
  Disable pylint too-many-nested-blocks in _RunCmdPipe
  Reduce nesting in import-export ProcessChildIO
  Reduce nesting in LUOobCommand.Exec
  Reduce nesting in LUInstanceCreate.RunOsScripts
  Reduce nesting in RemoveNodeSshKeyBulk key calculation
  Reduce nesting in RemoveNodeSshKeyBulk ssh logic
  Reduce nesting in gnt-cluster VerifyDisks missing disk loop
  Reduce nesting in _CheckVLANArguments
  Reduce nesting in StartDaemon
  Disable pylint bad-continuation warning
  Disable pylint superfluous-parens warning
  Disable pylint redefined-variable-type warning
  Disable pylint too-many-branches warnings
  Disable pylint broad-except warnings
  Disable incorrect pylint assigning-non-slot warning
  Quell pylint unbalanced-tuple-unpacking warning
  Cleanup: Use new-style classes everywhere
  Quell pylint socket.timeout warning
  Quell the pylint wrong-import-order warnings
  Quell cell-var-from-loop warning
  Use default value lambda param to avoid cell-var-from-loop
  Quell too-many-boolean-expressions
  Remove pylint tests removed in pylint 2.0
  Quell trailing newline
  Quell bad-whitespace warning
  Quell consider-using-enumerate warning
  Disable pylint unsubscriptable-object warning
  Disable pylint bare-except warning
  Disable unwanted pylint wrong-import-position warnings
  Disable pylint unused-wildcard-import warning
  Disable incorrect pylint not-callable warning
  Disable pylint unpacking-non-sequence warning
  Disable pylint misplaced-comparison-constant warning
  Disable incorect pylint simplify-if-statement warning
  Disable pylint eval-used warning
  Disable pylint invalid-name warning
  Disable pylint import-self warning
  Disable some pylint unused-import warnings
  Replace deprecated pylint >=0.27 pragma with new form
  Delete old warning disables removed from pylint 1.6
  Fix pylint >1.4 pycurl no-member warnings
  Cleanup: Remove unused/duplicate module/fn import
  Cleanup: Fix unidiomatic-typecheck
  Cleanup: Remove some unneeded pylint disables
  Cleanup: Iterate dict rather than key list
  Cleanup: Remove unused format key
  Cleanup: StartInstance and RebootInstance return None
  Cleanup: Fix for/else with no break in AddAuthorizedKeys
  Cleanup: Replace map/filters with list comprehensions
  Cleanup: del is a statement not a function
  Cleanup: Use FOO not in BAR instead of not FOO in BAR
  Cleanup: Simplify boolean assignment
  Cleanup: Remove some unnecessary if (...) parens
  Fix invalid variable error for file-based disks
  FIX: Refactor DiagnoseOS to use a loop, not an inner fn
  FIX: Set INSTANCE_NICn_NETWORK_NAME only if net is defined
  Fix for incorrect parsing of DRBD versions
  Change a few errors to report names, not UUIDs
  Give atomicWriteFile temp filenames a more distinct pattern
  LV check failure should print instance name
  Disable logging CallRPCMethod timings in non-debug configs
  568 Update hv_kvm to handle output from qemu >= 1.6.0

Trivial merge conflicts:
  lib/cli.py                   -- whitespace
  lib/cmdlib/cluster/verify.py -- code previously removed from 2.16
  lib/hypervisor/hv_xen.py     -- tuple vs list
  lib/luxi.py                  -- whitespace
  lib/server/masterd.py        -- code cleaned up in 2.15 and
                                  previously removed in 2.16
  lib/storage/filestorage.py   -- whitespace & generator instead of list
  lib/tools/node_cleanup.py    -- whitespace

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoRevert "Merge branch 'stable-2.15' into stable-2.16"
Brian Foley [Fri, 16 Dec 2016 16:57:37 +0000 (16:57 +0000)]
Revert "Merge branch 'stable-2.15' into stable-2.16"

This reverts commit 831523921beebc9fd607c9d78c7f0ee40bc074d4, reversing
changes made to da3f3000e59d03e60634c9c3092f6cc6e1e63389 (a mistaken
merge to stable-2.15 of what was intended to be a 2.15->2.16 merge)

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoMerge branch 'stable-2.15' into stable-2.16
Brian Foley [Fri, 16 Dec 2016 16:01:48 +0000 (16:01 +0000)]
Merge branch 'stable-2.15' into stable-2.16

Merge forward patches from stable-2.15

* stable-2.15
  Fix gnt-instance console instance unpausing for xl toolstack
  Disable pylint too-many-nested-blocks in _RunCmdPipe
  Reduce nesting in import-export ProcessChildIO
  Reduce nesting in LUOobCommand.Exec
  Reduce nesting in LUInstanceCreate.RunOsScripts
  Reduce nesting in RemoveNodeSshKeyBulk key calculation
  Reduce nesting in RemoveNodeSshKeyBulk ssh logic
  Reduce nesting in gnt-cluster VerifyDisks missing disk loop
  Reduce nesting in _CheckVLANArguments
  Reduce nesting in StartDaemon
  Disable pylint bad-continuation warning
  Disable pylint superfluous-parens warning
  Disable pylint redefined-variable-type warning
  Disable pylint too-many-branches warnings
  Disable pylint broad-except warnings
  Disable incorrect pylint assigning-non-slot warning
  Quell pylint unbalanced-tuple-unpacking warning
  Cleanup: Use new-style classes everywhere
  Quell pylint socket.timeout warning
  Quell the pylint wrong-import-order warnings
  Quell cell-var-from-loop warning
  Use default value lambda param to avoid cell-var-from-loop
  Quell too-many-boolean-expressions
  Remove pylint tests removed in pylint 2.0
  Quell trailing newline
  Quell bad-whitespace warning
  Quell consider-using-enumerate warning
  Disable pylint unsubscriptable-object warning
  Disable pylint bare-except warning
  Disable unwanted pylint wrong-import-position warnings
  Disable pylint unused-wildcard-import warning
  Disable incorrect pylint not-callable warning
  Disable pylint unpacking-non-sequence warning
  Disable pylint misplaced-comparison-constant warning
  Disable incorect pylint simplify-if-statement warning
  Disable pylint eval-used warning
  Disable pylint invalid-name warning
  Disable pylint import-self warning
  Disable some pylint unused-import warnings
  Replace deprecated pylint >=0.27 pragma with new form
  Delete old warning disables removed from pylint 1.6
  Fix pylint >1.4 pycurl no-member warnings
  Cleanup: Remove unused/duplicate module/fn import
  Cleanup: Fix unidiomatic-typecheck
  Cleanup: Remove some unneeded pylint disables
  Cleanup: Iterate dict rather than key list
  Cleanup: Remove unused format key
  Cleanup: StartInstance and RebootInstance return None
  Cleanup: Fix for/else with no break in AddAuthorizedKeys
  Cleanup: Replace map/filters with list comprehensions
  Cleanup: del is a statement not a function
  Cleanup: Use FOO not in BAR instead of not FOO in BAR
  Cleanup: Simplify boolean assignment
  Cleanup: Remove some unnecessary if (...) parens
  Fix invalid variable error for file-based disks
  FIX: Refactor DiagnoseOS to use a loop, not an inner fn
  FIX: Set INSTANCE_NICn_NETWORK_NAME only if net is defined
  Fix for incorrect parsing of DRBD versions
  Change a few errors to report names, not UUIDs
  Give atomicWriteFile temp filenames a more distinct pattern
  LV check failure should print instance name
  Disable logging CallRPCMethod timings in non-debug configs
  568 Update hv_kvm to handle output from qemu >= 1.6.0

Trivial merge conflicts:
  lib/cli.py                   -- whitespace
  lib/cmdlib/cluster/verify.py -- code previously removed from 2.16
  lib/hypervisor/hv_xen.py     -- tuple vs list
  lib/luxi.py                  -- whitespace
  lib/server/masterd.py        -- code cleaned up in 2.15 and
                                  previously removed in 2.16
  lib/storage/filestorage.py   -- whitespace & generator instead of list
  lib/tools/node_cleanup.py    -- whitespace

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoFix gnt-instance console instance unpausing for xl toolstack
Nicolas Avrutin [Thu, 15 Dec 2016 17:05:46 +0000 (09:05 -0800)]
Fix gnt-instance console instance unpausing for xl toolstack

xen-console-wrapper called the "xm" command instead of using the toolstack
configured in Ganeti. This patch replaces those calls with calls to the
toolstack command which is passed as the first argument to the script.

In addition, whereas "xm list --long" output a sexp, "xl list --long" outputs
JSON. Unfortunately, the JSON data is missing the "state" field, which we need
to determine whether the instance is paused. Thus we have to use
"xm list"/"xl list" and chop off the header line (because there is no flag
to omit printing the headers).

Signed-off-by: Nicolas Avrutin <rasputin@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

3 years agoDisable pylint too-many-nested-blocks in _RunCmdPipe
Brian Foley [Tue, 6 Dec 2016 14:46:32 +0000 (14:46 +0000)]
Disable pylint too-many-nested-blocks in _RunCmdPipe

There doesn't appear to be any easy way of reducing the complexity
of this function without moving it elsewhere or completely reorganising
the function, so disable this warning for the time being.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

3 years agoReduce nesting in import-export ProcessChildIO
Brian Foley [Tue, 6 Dec 2016 15:18:06 +0000 (15:18 +0000)]
Reduce nesting in import-export ProcessChildIO

This avoids some pylint too-many-nested-blocks warnings. Do this by
flattening some 'arrow antipattern code' inside the polling loop to use
guards instead.

This is hard to see from diff, but git difftool -y -x 'diff -y' will
show the before and after side by side.

Sprinkle some comments while we're here.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

3 years agoReduce nesting in LUOobCommand.Exec
Brian Foley [Tue, 6 Dec 2016 15:22:24 +0000 (15:22 +0000)]
Reduce nesting in LUOobCommand.Exec

This avoids a pylint too-many-nested-blocks warning.

NB it's hard to see from the diff because of all the whitespace, but
this just turns the if result.fail_msg check into a guard that continues
the next iteration of the loop, and dedends the rest of the loop body
rather than having it as an else:

git diff -w will show this.

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

3 years agoReduce nesting in LUInstanceCreate.RunOsScripts
Brian Foley [Tue, 6 Dec 2016 15:21:59 +0000 (15:21 +0000)]
Reduce nesting in LUInstanceCreate.RunOsScripts

This avoids a pylint too-many-nested-blocks warning.

NB it's hard to see from the diff because of all the whitespace, but
this just replaces the top "if iobj.disk and not self.adopt_disks" with
a pair of guards that return early, and dedents the rest of the fn.

git diff -w will show this.

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

3 years agoReduce nesting in RemoveNodeSshKeyBulk key calculation
Brian Foley [Tue, 6 Dec 2016 14:30:57 +0000 (14:30 +0000)]
Reduce nesting in RemoveNodeSshKeyBulk key calculation

This avoids a pylint too-many-nested-blocks warning. This removes the
6th level of nesting in the function, but may also be marginally faster
by turning the calculation into the set difference operation it really
is.

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

3 years agoReduce nesting in RemoveNodeSshKeyBulk ssh logic
Brian Foley [Tue, 6 Dec 2016 14:15:14 +0000 (14:15 +0000)]
Reduce nesting in RemoveNodeSshKeyBulk ssh logic

This avoids a pylint too-many-nested-blocks warning. It also removes
some copy & paste code, showing that the master candidate and ordinary
node case are the same apart from the logging.

No funtional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

3 years agoReduce nesting in gnt-cluster VerifyDisks missing disk loop
Brian Foley [Tue, 6 Dec 2016 15:19:01 +0000 (15:19 +0000)]
Reduce nesting in gnt-cluster VerifyDisks missing disk loop

To avoid pylint's too-many-nested-blocks warning.

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

3 years agoReduce nesting in _CheckVLANArguments
Brian Foley [Tue, 6 Dec 2016 15:19:50 +0000 (15:19 +0000)]
Reduce nesting in _CheckVLANArguments

This avoids a pylint too-many-nested-blocks warning. It also has the
happy side effect of removing some duplicated code.

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

3 years agoReduce nesting in StartDaemon
Brian Foley [Tue, 6 Dec 2016 15:22:36 +0000 (15:22 +0000)]
Reduce nesting in StartDaemon

This avoids a pylint too-many-nested-blocks warning.

The extra try: finally: os._exit(1) is unnecessary as _StartDaemonChild
already catches all its exceptions and if it ever finishes, calls
os._exit(1) anyways.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

3 years agoDisable pylint bad-continuation warning
Brian Foley [Mon, 5 Dec 2016 18:41:28 +0000 (18:41 +0000)]
Disable pylint bad-continuation warning

pylint is much more strict than pep8, and it would be too invasive
(and arguably pointless) to update these right now.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

3 years agoDisable pylint superfluous-parens warning
Brian Foley [Mon, 5 Dec 2016 18:20:13 +0000 (18:20 +0000)]
Disable pylint superfluous-parens warning

There are too many cases where we deliberately wrap expressions in
parens, either to indicate comparisons, or to allow multiline
expressions without line continuation chars, or to clarify confusing
precedence.

While here, clean up a few unpythonic cases.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

3 years agoDisable pylint redefined-variable-type warning
Brian Foley [Mon, 5 Dec 2016 17:29:30 +0000 (17:29 +0000)]
Disable pylint redefined-variable-type warning

There are a large number of cases where Ganeti assigns multiple types
(eg set/list, float/int) to the same variable at different times, and
although these would make a type checking tool very unhappy, they are
benign here (and besides, cleaning all this up would be too invasive).

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

3 years agoDisable pylint too-many-branches warnings
Brian Foley [Mon, 5 Dec 2016 11:39:36 +0000 (11:39 +0000)]
Disable pylint too-many-branches warnings

This is useful, but in some cases is a little too conservative. A fn
can have a lot of branches, but very little nesting, and can still be
easy to understand. This occurs in, eg, XenPvmHypervisor._GetConfig.

Ganeti has many functions that fail this check, so disable it for now
until we get a chance to clean up the codebase.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

3 years agoDisable pylint broad-except warnings
Brian Foley [Sun, 4 Dec 2016 21:01:56 +0000 (21:01 +0000)]
Disable pylint broad-except warnings

These are all deliberate top-level catch-any-unhandled-exception cases,
used for logging and error handling so just get pylint to ignore them.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

3 years agoDisable incorrect pylint assigning-non-slot warning
Brian Foley [Mon, 5 Dec 2016 11:07:27 +0000 (11:07 +0000)]
Disable incorrect pylint assigning-non-slot warning

This occurs pretty heavily in lib/objects.py, where pylint isn't
correctly detecting the __slots__ assignment. This appears to be
a known issue: https://github.com/PyCQA/pylint/issues/379

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

3 years agoQuell pylint unbalanced-tuple-unpacking warning
Brian Foley [Sun, 4 Dec 2016 20:55:26 +0000 (20:55 +0000)]
Quell pylint unbalanced-tuple-unpacking warning

Both of these functions return a list, not a tuple, and by manually
tracing the code, we can see they always return non-empty lists.

Change the callers to get the first element of the list rather than
using destructuring.

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

3 years agoCleanup: Use new-style classes everywhere
Brian Foley [Mon, 5 Dec 2016 16:56:58 +0000 (16:56 +0000)]
Cleanup: Use new-style classes everywhere

This quells pylint's old-style-class warning. For all classes changed by
this commit, none of the differences between new-style and old classes
matter: we don't subclass any of these classes, or use super()/type().

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

3 years agoQuell pylint socket.timeout warning
Brian Foley [Sun, 4 Dec 2016 19:12:53 +0000 (19:12 +0000)]
Quell pylint socket.timeout warning

socket.timeout doesn't have string or errno attributes.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoQuell the pylint wrong-import-order warnings
Brian Foley [Tue, 18 Oct 2016 19:46:38 +0000 (20:46 +0100)]
Quell the pylint wrong-import-order warnings

Sort the imports to what pylint thinks is correct.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoQuell cell-var-from-loop warning
Brian Foley [Tue, 18 Oct 2016 19:15:49 +0000 (20:15 +0100)]
Quell cell-var-from-loop warning

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoUse default value lambda param to avoid cell-var-from-loop
Brian Foley [Tue, 18 Oct 2016 19:11:04 +0000 (20:11 +0100)]
Use default value lambda param to avoid cell-var-from-loop

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoQuell too-many-boolean-expressions
Brian Foley [Tue, 18 Oct 2016 18:40:28 +0000 (19:40 +0100)]
Quell too-many-boolean-expressions

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoRemove pylint tests removed in pylint 2.0
Brian Foley [Tue, 18 Oct 2016 17:31:43 +0000 (18:31 +0100)]
Remove pylint tests removed in pylint 2.0

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoQuell trailing newline
Brian Foley [Tue, 18 Oct 2016 17:12:57 +0000 (18:12 +0100)]
Quell trailing newline

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoQuell bad-whitespace warning
Brian Foley [Mon, 17 Oct 2016 15:19:09 +0000 (16:19 +0100)]
Quell bad-whitespace warning

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoQuell consider-using-enumerate warning
Brian Foley [Mon, 17 Oct 2016 15:05:07 +0000 (16:05 +0100)]
Quell consider-using-enumerate warning

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoDisable pylint unsubscriptable-object warning
Brian Foley [Sun, 4 Dec 2016 20:42:41 +0000 (20:42 +0000)]
Disable pylint unsubscriptable-object warning

In these cases result is subscriptable.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoDisable pylint bare-except warning
Brian Foley [Sun, 4 Dec 2016 19:20:58 +0000 (19:20 +0000)]
Disable pylint bare-except warning

This is actually dead code which wasn't removed until Ganeti 2.16

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoDisable unwanted pylint wrong-import-position warnings
Brian Foley [Tue, 18 Oct 2016 19:51:41 +0000 (20:51 +0100)]
Disable unwanted pylint wrong-import-position warnings

For these, we want to import some modules in a particular order, so
disable the pylint warnings.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoDisable pylint unused-wildcard-import warning
Brian Foley [Tue, 18 Oct 2016 19:56:57 +0000 (20:56 +0100)]
Disable pylint unused-wildcard-import warning

We're deliberately importing all the variables defined in cli_opt into
cli's namespace.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoDisable incorrect pylint not-callable warning
Brian Foley [Tue, 18 Oct 2016 19:23:03 +0000 (20:23 +0100)]
Disable incorrect pylint not-callable warning

REQ_REQUEST will be a parameter validating function from ht, but pylint
can't statically determine this.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoDisable pylint unpacking-non-sequence warning
Brian Foley [Tue, 18 Oct 2016 19:21:46 +0000 (20:21 +0100)]
Disable pylint unpacking-non-sequence warning

For these cases, self.ia_result actually will have a tuple assigned to
it, it's just hard for static analysis tools to see it.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoDisable pylint misplaced-comparison-constant warning
Brian Foley [Tue, 18 Oct 2016 19:20:12 +0000 (20:20 +0100)]
Disable pylint misplaced-comparison-constant warning

It's correct, and easier to read a < x < b in this case.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoDisable incorect pylint simplify-if-statement warning
Brian Foley [Tue, 18 Oct 2016 19:01:04 +0000 (20:01 +0100)]
Disable incorect pylint simplify-if-statement warning

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoDisable pylint eval-used warning
Brian Foley [Mon, 17 Oct 2016 15:00:03 +0000 (16:00 +0100)]
Disable pylint eval-used warning

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoDisable pylint invalid-name warning
Brian Foley [Mon, 17 Oct 2016 14:07:56 +0000 (15:07 +0100)]
Disable pylint invalid-name warning

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoDisable pylint import-self warning
Brian Foley [Mon, 17 Oct 2016 14:07:43 +0000 (15:07 +0100)]
Disable pylint import-self warning

In this case it's a deliberate test to make sure that the contents
of lib/ have been correctly installed in a dir called ganeti.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoDisable some pylint unused-import warnings
Brian Foley [Mon, 17 Oct 2016 13:52:52 +0000 (14:52 +0100)]
Disable some pylint unused-import warnings

Technically the imports are unused in the module, but they're being
imported to make them available to users of the module.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoReplace deprecated pylint >=0.27 pragma with new form
Brian Foley [Sun, 4 Dec 2016 20:19:43 +0000 (20:19 +0000)]
Replace deprecated pylint >=0.27 pragma with new form

The disable-all pragma was deprecated in favour of the new skip-file
pragma in pylint >= 0.27. Since we're moving to 1.6.4, use this instead.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoDelete old warning disables removed from pylint 1.6
Brian Foley [Tue, 18 Oct 2016 20:37:33 +0000 (21:37 +0100)]
Delete old warning disables removed from pylint 1.6

These include star-args and r0924: badly implemented immutable.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoFix pylint >1.4 pycurl no-member warnings
Brian Foley [Sun, 4 Dec 2016 18:49:16 +0000 (18:49 +0000)]
Fix pylint >1.4 pycurl no-member warnings

Add pycurl to the whitelist of C extensions that can be loaded for the
purposes of attribute checking. This works around a security feature
introduced in pylint 1.4 to avoid arbitrary code injection.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoCleanup: Remove unused/duplicate module/fn import
Brian Foley [Tue, 18 Oct 2016 18:33:39 +0000 (19:33 +0100)]
Cleanup: Remove unused/duplicate module/fn import

No functional change. Found by pylint's reimported warning.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoCleanup: Fix unidiomatic-typecheck
Brian Foley [Tue, 18 Oct 2016 17:29:15 +0000 (18:29 +0100)]
Cleanup: Fix unidiomatic-typecheck

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoCleanup: Remove some unneeded pylint disables
Brian Foley [Tue, 18 Oct 2016 17:22:52 +0000 (18:22 +0100)]
Cleanup: Remove some unneeded pylint disables

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoCleanup: Iterate dict rather than key list
Brian Foley [Mon, 17 Oct 2016 16:49:50 +0000 (17:49 +0100)]
Cleanup: Iterate dict rather than key list

The later explicitly constructs a new list of the dictionary's keys, which
we don't need here. Quietens pylint's consider-iterating-dictionary warning.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoCleanup: Remove unused format key
Brian Foley [Mon, 17 Oct 2016 15:48:03 +0000 (16:48 +0100)]
Cleanup: Remove unused format key

No functional change, but quietens pylint unused-format-string-argument
warning.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoCleanup: StartInstance and RebootInstance return None
Brian Foley [Mon, 17 Oct 2016 15:22:14 +0000 (16:22 +0100)]
Cleanup: StartInstance and RebootInstance return None

Quell assignment-from-none warning

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoCleanup: Fix for/else with no break in AddAuthorizedKeys
Brian Foley [Sun, 4 Dec 2016 19:35:09 +0000 (19:35 +0000)]
Cleanup: Fix for/else with no break in AddAuthorizedKeys

After a refactoring in c178396, the only break statement in the for/else
loop in AddAuthorizedKeys was removed. This means the else block is
always executed, so remove the else: to quell a pylint warning.

No functional change.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoCleanup: Replace map/filters with list comprehensions
Brian Foley [Mon, 17 Oct 2016 14:54:27 +0000 (15:54 +0100)]
Cleanup: Replace map/filters with list comprehensions

No functional change, but more pythonic and quells pylint's
deprecated-lambda warning.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoCleanup: del is a statement not a function
Brian Foley [Mon, 17 Oct 2016 13:16:40 +0000 (14:16 +0100)]
Cleanup: del is a statement not a function

Remove unnecessary parens to make pylint happy.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoCleanup: Use FOO not in BAR instead of not FOO in BAR
Brian Foley [Mon, 17 Oct 2016 13:14:31 +0000 (14:14 +0100)]
Cleanup: Use FOO not in BAR instead of not FOO in BAR

No functional change, but more pythonic, and makes pylint's
unneeded-not warning happy.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoCleanup: Simplify boolean assignment
Brian Foley [Fri, 2 Dec 2016 17:14:32 +0000 (17:14 +0000)]
Cleanup: Simplify boolean assignment

No functional change. Found by pylint's simplifiable-if-statement.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoCleanup: Remove some unnecessary if (...) parens
Brian Foley [Fri, 2 Dec 2016 17:11:02 +0000 (17:11 +0000)]
Cleanup: Remove some unnecessary if (...) parens

No functional change, but makes pylint 1.1.0 a little happier, and
is more pythonic.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Federico Pareschi <morg@google.com>

3 years agoFix invalid variable error for file-based disks
Brian Foley [Mon, 17 Oct 2016 16:27:26 +0000 (17:27 +0100)]
Fix invalid variable error for file-based disks

Introduced by 94e252a33. Found by pylint's undefined-loop-variable
warning.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

3 years agoFIX: Refactor DiagnoseOS to use a loop, not an inner fn
Brian Foley [Mon, 17 Oct 2016 13:24:26 +0000 (14:24 +0100)]
FIX: Refactor DiagnoseOS to use a loop, not an inner fn

Found by pylint's cell-var-from-loop.

The inner function tries to access a dictionary that's local to the
outer function, assuming that Python has similar lexical scoping rules
to most languages. Unfortunately it doesn't. See
https://docs.python.org/3/tutorial/classes.html#python-scopes-and-namespaces

Refactor code to work as intended by moving the fn body into a loop.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

3 years agoFIX: Set INSTANCE_NICn_NETWORK_NAME only if net is defined
Brian Foley [Mon, 17 Oct 2016 13:08:32 +0000 (14:08 +0100)]
FIX: Set INSTANCE_NICn_NETWORK_NAME only if net is defined

Found by pylint's using-constant-test warning. 'network' is a module,
'net' is the network name for this NIC.

Signed-off-by: Brian Foley <bpfoley@google.com>
Reviewed-by: Viktor Bachraty <vbachraty@google.com>

3 years agoStartInstance restores instance state if running
Viktor Bachraty [Fri, 7 Oct 2016 19:10:34 +0000 (20:10 +0100)]
StartInstance restores instance state if running

Add a side effect to StartInstance on Xen hypervisor that restores the
Xen configuration in case it is missing. This is a common failure
scenario during migrations. The generic backend checks if the instance
is running and queries the hypervisor about it's state. If it needs
recovery, the backend gathers the associated block devices and calls
RestoreInstance in the same manner as if it would call StartInstance in
case the instance wasn't already running.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

3 years agoAllow migrate --cleanup to adopt an instance
Viktor Bachraty [Fri, 7 Oct 2016 17:57:22 +0000 (18:57 +0100)]
Allow migrate --cleanup to adopt an instance

In some failure modes, Ganeti state of record may desync from actual
state of world. This patch allows migrate --cleanup to adopt an instance
if it is detected running on an unexpected node.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

3 years agoMake migrate --cleanup more robust
Viktor Bachraty [Fri, 7 Oct 2016 16:29:23 +0000 (17:29 +0100)]
Make migrate --cleanup more robust

In case of Xen migrations, the most common failure case is when the
instance fails to freeze so the migration fails with domains running on
both target and source node. This patchs allows migrate --cleanup to
recover by running AbortMigrate() in case the instance is running on
both the source and target node.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>

3 years agoMake finalize_migration_{src,dst} a single op
Viktor Bachraty [Fri, 7 Oct 2016 16:17:44 +0000 (17:17 +0100)]
Make finalize_migration_{src,dst} a single op

FinalizeMigrationSource and FinalizeMigrationDst should compose an
single operation consisting of 2 idempotent steps, that can be always
both retried, similarly how they are used in AbortMigration() without
any reliance on the order of their execution. Otherwise we would have to
remember at which step did the migration fail in order to be able to
rollback/recover. Also state of the record should be updated only if
both succeeded, otherwise we won't know in case of recovery if the
record corresponds to the original state or the final state.

Signed-off-by: Viktor Bachraty <vbachraty@google.com>
Reviewed-by: Brian Foley <bpfoley@google.com>