ganeti-github.git
7 years agohv_xen: Test stopping an instance
Michael Hanselmann [Thu, 24 Jan 2013 14:09:38 +0000 (15:09 +0100)]
hv_xen: Test stopping an instance

Test stopping an instance, and ensure the configuration file still
exists after an instance couldn't be stopped.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

7 years agohv_xen: Test starting an instance
Michael Hanselmann [Thu, 24 Jan 2013 14:11:47 +0000 (15:11 +0100)]
hv_xen: Test starting an instance

Test starting an instance in normal mode, in paused mode, and in case of
failure.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

7 years agohv_xen: Test reading non-existent config file
Michael Hanselmann [Thu, 24 Jan 2013 14:11:09 +0000 (15:11 +0100)]
hv_xen: Test reading non-existent config file

Check whether doing so raises an exception.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

7 years agohv_xen: Test verifying hypervisor
Michael Hanselmann [Thu, 24 Jan 2013 14:10:37 +0000 (15:10 +0100)]
hv_xen: Test verifying hypervisor

Test success and failure.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

7 years agohv_xen: Test removing config from "auto" directory
Michael Hanselmann [Thu, 24 Jan 2013 14:10:11 +0000 (15:10 +0100)]
hv_xen: Test removing config from "auto" directory

Test whether any existing configuration file named after the instance is
removed from Xen's “auto” directory.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

7 years agoEstablish base for testing Xen hypervisor abstraction
Michael Hanselmann [Thu, 24 Jan 2013 14:08:50 +0000 (15:08 +0100)]
Establish base for testing Xen hypervisor abstraction

There are two separate Xen hypervisors (HVM and PVM), as well as two
different Xen commands (xl and xm). This already provides four different
combinations and future changes might bring even more. For this reason
the test classes for the Xen hypervisor are not defined manually, but
rather generated at module load time. Doing so makes it possible to keep
using the standard unit test infrastructure provided by the “unittest”
module.

Using “super”, which we try to avoid in other places, is necessary to
call functions provided by other classes.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

7 years agohv_xen: Test errors while writing config file
Michael Hanselmann [Thu, 24 Jan 2013 14:02:42 +0000 (15:02 +0100)]
hv_xen: Test errors while writing config file

Test error handling while writing Xen configuration files.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

7 years agohv_xen: Test using unknown Xen command
Michael Hanselmann [Thu, 24 Jan 2013 14:01:55 +0000 (15:01 +0100)]
hv_xen: Test using unknown Xen command

This tests “unknown” values of “constants.XEN_CMD”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

7 years agohv_xen: Split StopInstance
Michael Hanselmann [Thu, 24 Jan 2013 14:00:36 +0000 (15:00 +0100)]
hv_xen: Split StopInstance

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

7 years agohv_xen: Abstract running Xen commands
Michael Hanselmann [Tue, 22 Jan 2013 14:54:53 +0000 (15:54 +0100)]
hv_xen: Abstract running Xen commands

Instead of using the “XEN_CMD” constant in multiple places, that is now
all in a single place and can easily be changed for unit tests (through
a parameter given to the constructor).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

7 years agoMerge branch 'devel-2.7'
Michael Hanselmann [Fri, 25 Jan 2013 10:12:30 +0000 (11:12 +0100)]
Merge branch 'devel-2.7'

* devel-2.7:
  hv_xen: Remove config after shutdown was successful

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

7 years agoList networks: call client explicitely
Helga Velroyen [Thu, 24 Jan 2013 14:47:57 +0000 (15:47 +0100)]
List networks: call client explicitely

This way, it is easier to make it call the haskell
implementation of the network queries.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agoFirst part of Network Queries in Haskell
Helga Velroyen [Thu, 24 Jan 2013 14:51:35 +0000 (15:51 +0100)]
First part of Network Queries in Haskell

This is the beginning of the implementation of network
queries. This includes establishing all infrastructure
to run the network queries and implement querying of
some simpler fields and the node group listing.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agoConvert Maybe results to RSUnavail
Helga Velroyen [Thu, 24 Jan 2013 14:50:36 +0000 (15:50 +0100)]
Convert Maybe results to RSUnavail

When displaying query results of type Maybe, one could use the
function rsMaybe. Unfortunately, this function maps 'Nothing'
values to RSNoData which gets displayed as '?' in the list of
query results. These semantics do not fit if the result is of
a Maybe type, because it is an optional field, because in this
case 'Nothing' is not an error (like for example the consequence
of an RPC error), but a legitimate result and as such should
be displayed as '-' in the result list. For this purpose,
this patch introduces another version of rsMaybe, which is
called rsMaybeUnavail to be used for optional fields.
For consistency, rsMaybe gets renamed to rsMaybeNoData.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agoExtend config by networks and networks by UUIDs
Helga Velroyen [Thu, 24 Jan 2013 14:46:41 +0000 (15:46 +0100)]
Extend config by networks and networks by UUIDs

For network queries to work, we need to extend the general
config type to include all available networks. Additionally,
we add UUIDs to the network type itself.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agoTypo in comment of network type
Helga Velroyen [Thu, 24 Jan 2013 15:54:34 +0000 (16:54 +0100)]
Typo in comment of network type

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agohv_xen: Remove config after shutdown was successful
Michael Hanselmann [Thu, 24 Jan 2013 13:29:18 +0000 (14:29 +0100)]
hv_xen: Remove config after shutdown was successful

If stopping an instance failed, the configuration would already be gone
and other operations depending on it (e.g. migration) would no longer
work. With this patch the configuration file is only removed once the
instance was succesfully stopped or destroyed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

7 years agohv_xen: Simplify writing configuration
Michael Hanselmann [Tue, 22 Jan 2013 14:42:07 +0000 (15:42 +0100)]
hv_xen: Simplify writing configuration

Instead of calling _WriteConfigFileStatic from both derived classes,
those now only return the content (minus the “do not edit” header). The
configuration is then written by the base class, XenHypervisor.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agohv_xen: Prepare for unit tests, remove {static,class}method
Michael Hanselmann [Tue, 22 Jan 2013 13:27:48 +0000 (14:27 +0100)]
hv_xen: Prepare for unit tests, remove {static,class}method

Unit tests will have to specify a custom Xen configuration directory (a
temporary directory). To this end “hv_xen.XenHypervisor” and its two
derived classes are modified to have fewer static and class methods. A
new private attribute for the configuration directory is also added (to
be used in test code).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

7 years agohv_xen: Factorize and test disk configuration
Michael Hanselmann [Tue, 22 Jan 2013 12:31:25 +0000 (13:31 +0100)]
hv_xen: Factorize and test disk configuration

The “_GetConfigFileDiskData” function is moved to module level and
cleaned up (module-level constants for letters and file I/O drivers).

Until now only 24 disks would be supported (e.g. “sda” to “sdx”), when
the Latin alphabet actually has 26 characters. Now all 26 letters would
be available for use (“constants.MAX_DISKS” is still 16).

Newly added unit tests provide complete coverage for
“_GetConfigFileDiskData”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

7 years agohv_xen: Refactor getting node information, add tests
Michael Hanselmann [Mon, 21 Jan 2013 17:17:42 +0000 (18:17 +0100)]
hv_xen: Refactor getting node information, add tests

Refactor and add tests for getting node (Domain-0) information.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

7 years agohv_xen: Refactor running & parsing "xm list", add tests
Michael Hanselmann [Fri, 18 Jan 2013 14:30:05 +0000 (15:30 +0100)]
hv_xen: Refactor running & parsing "xm list", add tests

This patch refactors “_RunXmList” and adds some tests.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

7 years agoMerge branch 'devel-2.7'
Michael Hanselmann [Thu, 24 Jan 2013 11:46:15 +0000 (12:46 +0100)]
Merge branch 'devel-2.7'

* devel-2.7:
  hv_*: Always return from Verify, style fixes
  _VerifyErrors()._Error() and _ErrorIf() are now consistent
  Handle the result of QueryGroups() correctly

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

7 years agohv_*: Always return from Verify, style fixes
Michael Hanselmann [Wed, 23 Jan 2013 17:41:34 +0000 (18:41 +0100)]
hv_*: Always return from Verify, style fixes

Change all “Verify” methods in hypervisor abstractions to explicitely
return None if no problem was detected. Remove punctuation from error
messages. Update docstrings with “@return” and some small mistakes.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agoCheck minimum size of networks on creation
Helga Velroyen [Mon, 21 Jan 2013 15:20:34 +0000 (16:20 +0100)]
Check minimum size of networks on creation

When creating a network, so far no size constraints were checked.
We now limit the size of a network to a /30 or bigger, although
tecnically, the ipaddr library supports even /32 networks.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agoNetwork QA
Helga Velroyen [Tue, 15 Jan 2013 14:34:55 +0000 (15:34 +0100)]
Network QA

This adds a script for the QA of 'gnt-network'. So far it
covers adding/removing and connecting/disconnecting networks.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agoFix syntax error in QA's sample config
Helga Velroyen [Mon, 21 Jan 2013 09:33:35 +0000 (10:33 +0100)]
Fix syntax error in QA's sample config

The additional comma makes this file invalid JSON.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agoconstants: Add list of known Xen commands
Michael Hanselmann [Wed, 23 Jan 2013 17:41:01 +0000 (18:41 +0100)]
constants: Add list of known Xen commands

These will be used in unittests.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years ago_VerifyErrors()._Error() and _ErrorIf() are now consistent
Bernardo Dal Seno [Mon, 21 Jan 2013 21:43:24 +0000 (22:43 +0100)]
_VerifyErrors()._Error() and _ErrorIf() are now consistent

_Error() didn't contain the logic for demoting errors to warnings and for
marking an operation as failed. Now _ErrorIf() is just a minimal wrapper
for _Error().

Unit tests included.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agoMerge branch 'master' into devel-2.7
Michael Hanselmann [Wed, 23 Jan 2013 13:37:42 +0000 (14:37 +0100)]
Merge branch 'master' into devel-2.7

* master:
  Handle the result of QueryGroups() correctly

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agoHandle the result of QueryGroups() correctly
Dimitris Aragiorgis [Tue, 22 Jan 2013 13:58:32 +0000 (15:58 +0200)]
Handle the result of QueryGroups() correctly

If no group is given for the “gnt-network connect“/“… disconnect”
commands, the client uses the result of “QueryGroups()” which is a list
of lists. Use “itertools.chain()” to handle the return value correctly.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
(cherry picked from commit af3aea481c3b8528239770edcae75d406a57de81)

7 years agoHandle the result of QueryGroups() correctly
Dimitris Aragiorgis [Tue, 22 Jan 2013 13:58:32 +0000 (15:58 +0200)]
Handle the result of QueryGroups() correctly

If no group is given for the “gnt-network connect“/“… disconnect”
commands, the client uses the result of “QueryGroups()” which is a list
of lists. Use “itertools.chain()” to handle the return value correctly.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agohv_xen: Compose file name outside error handling
Michael Hanselmann [Tue, 22 Jan 2013 13:25:52 +0000 (14:25 +0100)]
hv_xen: Compose file name outside error handling

In _ReadConfigFile, the filename should be prepared outside the
try/except block. Fixes bad code formatting, too.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

7 years agotestutils: Module-level method for reading test data
Michael Hanselmann [Tue, 22 Jan 2013 15:41:24 +0000 (16:41 +0100)]
testutils: Module-level method for reading test data

This patch moves the functions “_ReadTestData” and “_TestDataFilename”
to module level. They do not depend on a class instance at all.

Some cases of “utils.ReadFile(self._TestDataFilename(…))” are also
corrected.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

7 years agoAdd test for backend._GetBlockDevSymlinkPath
Michael Hanselmann [Tue, 22 Jan 2013 12:50:57 +0000 (13:50 +0100)]
Add test for backend._GetBlockDevSymlinkPath

Add a unit test for the trivial “_GetBlockDevSymlinkPath” function in
backend (small changes in the function were required).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

7 years agohv_base: Remove empty constructor
Michael Hanselmann [Tue, 22 Jan 2013 12:55:33 +0000 (13:55 +0100)]
hv_base: Remove empty constructor

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

7 years agoFix format string of KVM output
Helga Velroyen [Mon, 21 Jan 2013 15:12:04 +0000 (16:12 +0100)]
Fix format string of KVM output

This fixes a missing 's' in the format string and
the wrong quotes. Those bugs were introduced in
commit 6e043e60.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agoRenames and cleanup of variable names in confd
Iustin Pop [Thu, 17 Jan 2013 12:13:02 +0000 (13:13 +0100)]
Renames and cleanup of variable names in confd

The current names are quite confusing; this patch cleans up the
confusion by making sure we use different terms for the two threads,
etc.

No actual code changes besides the renames.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agoChange the polling thread sleep behaviour
Iustin Pop [Thu, 17 Jan 2013 12:04:13 +0000 (13:04 +0100)]
Change the polling thread sleep behaviour

Currently, the polling thread sleeps at the end of its body; the
original reason was that when the thread initially starts, we want to
load the config immediately (in the old-style config load model).

However, now we explicitly load the config, so we can make this thread
behave like the watcher thread: sleep first, then do the work. This
model improves for example the startup of the thread, by not redoing a
config check immediately after switching to polling mode.

We also increase the max idle poll rounds interval from 2 to 3; the
new behaviour means that we revert from polling to inotify after
around one second of no changes.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agoFirst part of confd timer changes
Iustin Pop [Thu, 17 Jan 2013 11:22:40 +0000 (12:22 +0100)]
First part of confd timer changes

This patch changes the resolution of the timers: the watcher timer
goes from 60s to 17s, and the polling-mode timer goes from 2 seconds
to 250ms. The code changes a bit more due to the changes in the units
of the various constants.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agoIncrease Haskell log file resolution
Iustin Pop [Thu, 17 Jan 2013 11:15:39 +0000 (12:15 +0100)]
Increase Haskell log file resolution

Currently the Haskell logging goes with the default time formatting,
which lacks sub-second resolution. This is not good, as it's too
coarse.

The patch adds picoseconds to the log; they will contain usually too
many zeroes, but the other option is to use a log format that (for
precisely zero fractional part) omits the separator, so it's not
consistent from line to line, and which also uses a dot as a
separator, different from the Python code.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agoAdd a function to get the current time in microseconds
Iustin Pop [Thu, 17 Jan 2013 10:38:45 +0000 (11:38 +0100)]
Add a function to get the current time in microseconds

In some cases we need higher resolution that seconds; I've settled on
microseconds as that is what 'threadDelay' wants, for exactly, so it's
easier if we keep the same units.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

7 years agoStart confd in notify mode if we can
Iustin Pop [Wed, 16 Jan 2013 16:32:00 +0000 (17:32 +0100)]
Start confd in notify mode if we can

This patch changes the default model—where confd is always started in
polling mode—to a model where if possible we enable inotify mode
before starting any of the background threads.

There are some tricky details here: if we enable inotify, we should
not modify the server state after wards, as inotify events could have
already fired and took ownership; therefore we presume that inotify
will work and set reloadModel=ReloadNotify, and only if we fail to
enable it we change it back to polling mode.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agoExplicitly load the config at confd startup
Iustin Pop [Wed, 16 Jan 2013 14:28:40 +0000 (15:28 +0100)]
Explicitly load the config at confd startup

Currently, confd main function doesn't load the config, but simply
forks the individual timer threads, which will—as a side effect-also
load the configuration as soon as they start.

However, this makes codes to the startup more complex, so let's
explicitly (try to) load the configuration as first thing.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agoFix type of 'node_whitelist' request parameter
Christos Stavrakakis [Fri, 18 Jan 2013 14:44:13 +0000 (16:44 +0200)]
Fix type of 'node_whitelist' request parameter

If opportunistic_locking is used, then 'node_whitelist' parameter passed
to the allocator is set to the LU's owned node locks. However, LU owned_locks
has type of 'set' while IReqInstanceAlloc expects type of
'ht.TMaybeListOf(ht.TNonEmptyString)'.

Signed-off-by: Christos Stavrakakis <cstavr@grnet.gr>
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agohv_xen: Add test for CPU pinning configuration
Michael Hanselmann [Fri, 18 Jan 2013 14:30:41 +0000 (15:30 +0100)]
hv_xen: Add test for CPU pinning configuration

Add a unittest for a function formatting CPU pinning information for
Xen's configuration.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agokvm: remove last version-based feature detection
Guido Trotter [Fri, 18 Jan 2013 14:06:33 +0000 (15:06 +0100)]
kvm: remove last version-based feature detection

This was left behind because it required a different kvm invocation.
Now that we can add new ones cheaply (two constants) it's easy to get
rid of it. Differently than in other cases we support old version which
don't have the actual option, and error out when it's invoked.

This also removes two hardcoded strings, and brings them at constant
level. A lot more are present, but two is megl che one (cit).

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agokvm: extract a regexp matching out of a for loop
Guido Trotter [Fri, 18 Jan 2013 14:06:33 +0000 (15:06 +0100)]
kvm: extract a regexp matching out of a for loop

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agokvm: deduplicate 'get output' code
Guido Trotter [Fri, 18 Jan 2013 13:54:08 +0000 (14:54 +0100)]
kvm: deduplicate 'get output' code

We had the same code twice, and were about to add a third time. Better
to collapse it into just one function.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agoMake Xen config path a build-time option
Michael Hanselmann [Fri, 18 Jan 2013 12:57:50 +0000 (13:57 +0100)]
Make Xen config path a build-time option

Stop hardcoding the path in “hv_xen.py”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agoMakefile.am: Protect placeholders for sed
Michael Hanselmann [Fri, 18 Jan 2013 13:00:04 +0000 (14:00 +0100)]
Makefile.am: Protect placeholders for sed

If the placeholder has exactly the same name as the variable used in
autoconf it would be replaced as well. In the case of
“@ENABLE_SPLIT_QUERY@” this led to the sed command “s#False#False#g”.
Inserting additional quotes after the first @ sign prevents this.

Since ENABLE_SPLIT_QUERY was not actually used it is removed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

7 years agoburnin: Don't keep hypervisor class around
Michael Hanselmann [Fri, 18 Jan 2013 13:13:56 +0000 (14:13 +0100)]
burnin: Don't keep hypervisor class around

Just determine whether it can migrate and keep that value instead of the
full hypervisor class.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agoText wrap in Haskell code
Yiannis Tsiouris [Wed, 16 Jan 2013 15:02:16 +0000 (17:02 +0200)]
Text wrap in Haskell code

This adds two functions in Utils.hs: 'wrap' and 'trim' along with some
simple tests. 'wrap' is used in Common.hs for wrapping command
descriptions to a fixed line length, e.g. useful for the "--help"
option.

Signed-off-by: Yiannis Tsiouris <gtsiour@softlab.ntua.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agoRun pre-migrate hooks on primary node too
Constantinos Venetsanopoulos [Thu, 17 Jan 2013 15:49:13 +0000 (17:49 +0200)]
Run pre-migrate hooks on primary node too

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agoMakefile: Allow tarball creation when man-in-doc is active
Michael Hanselmann [Thu, 17 Jan 2013 14:41:09 +0000 (15:41 +0100)]
Makefile: Allow tarball creation when man-in-doc is active

Documentation with included man pages should not be distributed as it
contains fixed paths. Since the documentation with man pages is actually
written to a separate directory, tarball creation is still possible.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agoCheck if KVM machine version is supported
Dimitris Aragiorgis [Wed, 16 Jan 2013 17:24:44 +0000 (19:24 +0200)]
Check if KVM machine version is supported

If machine version is passed as an hv param, a check is made
in target node whether this version is included in the supported
ones derived from kvm -M ? command.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agoRevert "run-in-tempdir: Improve speed"
Michael Hanselmann [Wed, 16 Jan 2013 17:34:05 +0000 (18:34 +0100)]
Revert "run-in-tempdir: Improve speed"

This reverts commit dd60179dfd8 as it broke coverage for Python.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agoVerify that templates are compatible with exclusive storage
Bernardo Dal Seno [Sun, 13 Jan 2013 17:47:27 +0000 (18:47 +0100)]
Verify that templates are compatible with exclusive storage

cluster-verify reports instances with disk templates not compatible with
exclusive storage but that are running on nodes with the exclusive storage
flag set.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agocluster-verify checks that PVs are not shared
Bernardo Dal Seno [Sun, 13 Jan 2013 15:37:11 +0000 (16:37 +0100)]
cluster-verify checks that PVs are not shared

When exclusive_storage is set, cluster-verify complains if PVs are shared
among unrelated LVs.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agobdev.GetPVInfo() returns list of LVs
Bernardo Dal Seno [Sun, 13 Jan 2013 14:54:33 +0000 (15:54 +0100)]
bdev.GetPVInfo() returns list of LVs

This will be used for checks related to exclusive storage.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agoMoved checks within LUClusterVerifyGroup
Bernardo Dal Seno [Fri, 11 Jan 2013 16:45:31 +0000 (17:45 +0100)]
Moved checks within LUClusterVerifyGroup

Almost all instance-specific checks have been moved from the Exec method to
the _VerifyInstance method. This cleans up Exec, which was becoming too big
even for pylint…

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agoadd-node checks PVs
Bernardo Dal Seno [Fri, 7 Dec 2012 18:19:02 +0000 (19:19 +0100)]
add-node checks PVs

add-node now performs a few checks on LVM before adding a node to the
cluster.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agoNew RPC call for node verify in dns-only RPC client
Bernardo Dal Seno [Mon, 10 Dec 2012 03:58:55 +0000 (04:58 +0100)]
New RPC call for node verify in dns-only RPC client

This will be used in node-add.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agocluster-verify checks uniformity of PV sizes
Bernardo Dal Seno [Thu, 6 Dec 2012 17:01:11 +0000 (18:01 +0100)]
cluster-verify checks uniformity of PV sizes

When exclusive_storage is set, cluster-verify complains if the size of PVs
is not uniform across a node group.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agoutils.LvmExclusiveCheckNodePvs() returns more info
Bernardo Dal Seno [Tue, 8 Jan 2013 20:54:29 +0000 (21:54 +0100)]
utils.LvmExclusiveCheckNodePvs() returns more info

utils.LvmExclusiveCheckNodePvs() now returns the size of the smallest and
biggest PVs of a node. They will be used for cross-node consistency checks.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agocluster-verify checks uniformity of exclusive_storage flag
Bernardo Dal Seno [Sun, 13 Jan 2013 15:07:19 +0000 (16:07 +0100)]
cluster-verify checks uniformity of exclusive_storage flag

The value of the flag should be the same for nodes within the same group.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agoSmall cleanup in constants.py
Bernardo Dal Seno [Mon, 7 Jan 2013 16:10:06 +0000 (17:10 +0100)]
Small cleanup in constants.py

- compat.UniqueFrozenset used in one more place, for consistency.
- The order of NV_XXX constants is fixed so that all values are now sorted
by name.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agorun-in-tempdir: Improve speed
Michael Hanselmann [Wed, 16 Jan 2013 15:40:54 +0000 (16:40 +0100)]
run-in-tempdir: Improve speed

By creating symlinks instead of actually creating files we still get the
benefits of using a temporary directory, but the preparation is about
three times as fast. Test used:

$ time bash -c 'for ((i=0; i < 100; ++i)); do
    autotools/run-in-tempdir true; done'

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

7 years agorun-in-tempdir: Copy doc/* only if needed
Michael Hanselmann [Wed, 16 Jan 2013 15:16:35 +0000 (16:16 +0100)]
run-in-tempdir: Copy doc/* only if needed

To preserve disk space on copying all files to a temporary directory,
the “doc” directory is only copied when building documentation. A
comment describing the need is also added to Makefile.am, as well as a
superfluous direct invocation of Bash removed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agoIntegrate DRBD with the report generation infrastructure
Michele Tartara [Tue, 8 Jan 2013 14:18:52 +0000 (15:18 +0100)]
Integrate DRBD with the report generation infrastructure

With this commit, the DRBD data collector does not only print the data
it extracts, but includes them in the proper JSON structure common to
all the data collectors, as prescribed by the design document.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agoAdd reporting infrastructure for data collectors
Michele Tartara [Tue, 8 Jan 2013 14:12:21 +0000 (14:12 +0000)]
Add reporting infrastructure for data collectors

This commit adds the part of the JSON report generation code that will be
common to all the data collectors, according to the format specified in the
design document.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agoMove getCurrentTime from Confd.Utils to Utils
Michele Tartara [Wed, 16 Jan 2013 13:03:01 +0000 (13:03 +0000)]
Move getCurrentTime from Confd.Utils to Utils

The getCurrentTime function is of wide interest, so its moved to a more
appropriate position.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agoburnin: Add unittest for supported disk templates
Michael Hanselmann [Tue, 15 Jan 2013 15:31:49 +0000 (16:31 +0100)]
burnin: Add unittest for supported disk templates

When new a new disk template is added, burnin should be updated.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agokvm: make kvm path configurable as an hv parameter
Guido Trotter [Mon, 14 Jan 2013 08:54:08 +0000 (08:54 +0000)]
kvm: make kvm path configurable as an hv parameter

This fixes Issue 225.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agomake _{Generate,Execute}KVMRuntime accept kvm help
Guido Trotter [Mon, 14 Jan 2013 08:45:41 +0000 (08:45 +0000)]
make _{Generate,Execute}KVMRuntime accept kvm help

Currently kvm --help is called twice at instance startup, this makes
sure it's called only once.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agokvm: make auxiliary functions accept a kvm path
Guido Trotter [Mon, 14 Jan 2013 08:38:05 +0000 (08:38 +0000)]
kvm: make auxiliary functions accept a kvm path

This will be used to be able to customize the path in an hv parameter.

Also fix two comments that still mentioned version-based checking.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

7 years agoAdd KVM machine version as hv param
Dimitris Aragiorgis [Tue, 15 Jan 2013 01:56:48 +0000 (03:56 +0200)]
Add KVM machine version as hv param

This is needed in case an instance must be booted with an exact type of
machine version (due to e.g. outdated drivers). If this hv param
explicitly defined (during instance creation/modification) then the kvm
process will be started having this version for the -M option. Otherwise
the default (derived from _GetDefaultMachineVersion() will be used.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agoMove burnin to ganeti.tools.burnin
Michael Hanselmann [Tue, 15 Jan 2013 15:35:06 +0000 (16:35 +0100)]
Move burnin to ganeti.tools.burnin

Similar to other scripts, “tools/burnin” just becomes a wrapper script.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agoRemove special-case of "0" in niceSort
Iustin Pop [Tue, 15 Jan 2013 16:24:43 +0000 (17:24 +0100)]
Remove special-case of "0" in niceSort

Since now niceSort is more consistently behaved, we don't have the
special casing. This should have been removed in commit a7f0953a, but
I didn't realise it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agoburnin: Enable DT_EXT/DT_RBD for single node clusters
Michael Hanselmann [Tue, 15 Jan 2013 15:31:38 +0000 (16:31 +0100)]
burnin: Enable DT_EXT/DT_RBD for single node clusters

As reported by Constantinos Venetsanopoulos, DT_RBD and DT_EXT work on
single node clusters.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agotest/Types.hs: add an HLint ANN for ignoring camelCase
Dato Simó [Tue, 15 Jan 2013 14:03:29 +0000 (14:03 +0000)]
test/Types.hs: add an HLint ANN for ignoring camelCase

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agoLink man pages in documentation
Michael Hanselmann [Tue, 8 Jan 2013 15:21:29 +0000 (16:21 +0100)]
Link man pages in documentation

This patch depends on “Option to include man pages in documentation”. In
the documentation build including man pages, all “:manpage:`…`”
references are converted to links. For man pages not provided by Ganeti,
Sphinx' standard formatting is used.

A small dance is necessary to hook into Sphinx' processing of man page
roles and to generate automatically resolved links. The code converts
“:manpage:`…`” for known man pages to the data structure equivalent of
“:doc:`$name($section) <man-$name>`”. Additionally it checks the section
numbers and formatting of references (in all builds).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agoMake niceSort a wrapper over niceSortKey id
Iustin Pop [Mon, 14 Jan 2013 14:16:28 +0000 (15:16 +0100)]
Make niceSort a wrapper over niceSortKey id

Currently, niceSort behaves differently than niceSortKey in cases
where the parsed groups are equal but the original not, as it uses
both members of the tuple as opposed to just the first one. Both
niceSortKey and the Python code use only the first one, so in corner
cases we can have different behaviour of these functions.

To rectify this, we change niceSort to be a simple wrapper over
"niceSortKey id", so that we get consistent behaviour.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agoAdd version check for Sphinx
Michael Hanselmann [Mon, 14 Jan 2013 15:16:10 +0000 (16:16 +0100)]
Add version check for Sphinx

Sphinx 1.0 and above can check version by setting “needs_sphinx” in the
configuration.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agoOption to include man pages in documentation
Michael Hanselmann [Wed, 9 Jan 2013 11:10:57 +0000 (12:10 +0100)]
Option to include man pages in documentation

Before this patch, HTML versions of man pages (man/*.rst) were already
built. However, since they are separate from the normal documentation,
their content is not indexed for Sphinx' search functionality.
Additionally it would simply be nice to have everything in one place.

To this end a new configure-time option is added to enable the inclusion
of man pages into the documentation. A dedicated option is necessary to
still be able to provide a static documentation build in the tarball
(not including man pages) as man pages contain build-specific paths and
values. The documentation with man pages is written to the directory
“doc/man-html”.

A future patch will extend Sphinx to link occurences of “:manpage:`…`”
to these man pages.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agodesign-autorepair.rst: separate list of jobs with plus sign
Dato Simó [Mon, 14 Jan 2013 20:41:46 +0000 (20:41 +0000)]
design-autorepair.rst: separate list of jobs with plus sign

Commas are not valid characters in tags, hence they can't be use to
separate the different job IDs; plus signs (+) are available, and not too
bad.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agoburnin: Disk template option cleanup
Michael Hanselmann [Tue, 15 Jan 2013 11:07:18 +0000 (12:07 +0100)]
burnin: Disk template option cleanup

Burnin doesn't support all disk templates (specifically it doesn't
support “blockdev”). The code, however, used manually composed lists in
some places, constants.DISK_TEMPLATES in others, and an internal list of
supported templates in some.

This patch changes burnin to only use its internal list of supported
disk templates.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agoburnin: Factorize disk template lists
Michael Hanselmann [Tue, 15 Jan 2013 10:59:16 +0000 (11:59 +0100)]
burnin: Factorize disk template lists

Keep the lists at the top instead of having them embedded in the code.
Use frozenset, too.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agoReplace nodegroups' PartialNic by PartialNicParams
Helga Velroyen [Thu, 10 Jan 2013 17:26:53 +0000 (18:26 +0100)]
Replace nodegroups' PartialNic by PartialNicParams

This fixes a bug that corrups the config when one adds a
network to a node group. I wil soon extend QA by some commands
that would have detected this.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

7 years agoLoader.hs: ignore expired ArSuspended policies
Dato Simó [Wed, 9 Jan 2013 18:12:14 +0000 (18:12 +0000)]
Loader.hs: ignore expired ArSuspended policies

At the moment, because 'mergeData' is pure, it may set instance auto-repair
policies that are of the form `ArSuspended $ Until timestamp_in_the_past`.
If later on the auto-repair tool notices this, it has lost access to what
would be the next-in-line policy to use (and would have to re-parse all
tags again).

To avoid this, we pass the current time to 'mergeData' from ExtLoader.hs,
and we propagate it to Loader.getArPolicy. ExtLoader.loadExternalData is in
the IO monad, so it has ready access to getClockTime.

A few other places were calling 'mergeData' directly. For Hscan.hs and
IAlloc.hs, we add appropriate calls to getClockTime. For files under test/,
we use a current time of 0.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agoLoader.hs: set instance auto-repair policy in mergeData
Dato Simó [Wed, 12 Dec 2012 12:45:37 +0000 (12:45 +0000)]
Loader.hs: set instance auto-repair policy in mergeData

'getArPolicy' and 'setArPolicy' follow the precedence rules introduced in
b1eb71c: within an object, the most restrictive tag wins; across object,
the nearest tag wins.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agodesign-autorepair.rst: clarify tag precedence and conflict
Dato Simó [Thu, 13 Dec 2012 19:29:14 +0000 (19:29 +0000)]
design-autorepair.rst: clarify tag precedence and conflict

This commit clarifies one particular point of the auto-repair workflow:
what to do when multiple, conflicting administrator-set tags exist in an
object; and how tags at different levels (cluster, node group and instance)
interact.

For conflict within an object, we choose to always let the most restrictive
tag win (i.e. the least destructive repair, and the longest suspension
time). For tags at different levels, we follow a simple "nearest tag wins"
rule.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agoLoader.hs: rewrite extractExTags to use chompPrefix
Dato Simó [Wed, 12 Dec 2012 12:42:15 +0000 (12:42 +0000)]
Loader.hs: rewrite extractExTags to use chompPrefix

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agoUtils.hs: function to chomp prefix + separator from a string
Dato Simó [Wed, 10 Oct 2012 20:08:03 +0000 (21:08 +0100)]
Utils.hs: function to chomp prefix + separator from a string

Some auto-repair tags are composed of a prefix and then an optional
argument; the new "chompPrefix" function in Utils.hs allows to strip a
prefix, allowing the last character (the separator) to be absent if there
is no argument.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agoUtils.hs: add a clockTimeToString function
Dato Simó [Fri, 7 Dec 2012 10:56:35 +0000 (10:56 +0000)]
Utils.hs: add a clockTimeToString function

This function allows to easily convert a ClockTime object to a string
representation of its timestamp (seconds-only).

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agoInstance.hs: add an 'arPolicy' field for auto-repair policy
Dato Simó [Fri, 7 Dec 2012 10:19:37 +0000 (10:19 +0000)]
Instance.hs: add an 'arPolicy' field for auto-repair policy

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agodesign-autorepair.rst: add a graph with state transitions
Dato Simó [Fri, 14 Dec 2012 22:13:38 +0000 (22:13 +0000)]
design-autorepair.rst: add a graph with state transitions

The graph added here follows the data types introduced in 7ec2f76, by which
"Suspended" is not a state, but a possible value of the auto-repair policy.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agoHTools/Types.hs: more auto-repair types
Dato Simó [Wed, 12 Dec 2012 12:45:27 +0000 (12:45 +0000)]
HTools/Types.hs: more auto-repair types

AutoRepairPolicy, AutoRepairStatus, and other auxiliary types are added.
These are used only internally by the auto-repair tool, and parsed from the
various object tags as defined in the design doc.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agoAdd initial constants and Haskell ADTs for auto repair
Dato Simó [Fri, 16 Nov 2012 22:16:10 +0000 (22:16 +0000)]
Add initial constants and Haskell ADTs for auto repair

In this commit, the AutoRepairType and AutoRepairResult types are defined,
with the possible values specified in doc/design-autorepair.rst.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

7 years agodesign-autorepair.rst: use the same prefix everywhere
Dato Simó [Thu, 6 Dec 2012 20:08:52 +0000 (20:08 +0000)]
design-autorepair.rst: use the same prefix everywhere

This commit changes the "ganeti:watcher:repair:pending" and
"ganeti:watcher:repair:result" tag prefixes to use
"ganeti:watcher:autorepair:{pending,result}", so that the same prefix is
used in all cases.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>