ganeti-github.git
6 years agoXen handle domain shutdown
Jose A. Lopes [Tue, 22 Oct 2013 16:28:41 +0000 (18:28 +0200)]
Xen handle domain shutdown

Update Xen backend to properly recognize when a domain has been
shutdown by the user and to properly cleanup a shutdown domain when
Ganeti requests Xen to stop this domain.

Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

6 years agoAdd instance state 'USER_down'
Jose A. Lopes [Tue, 22 Oct 2013 16:25:46 +0000 (18:25 +0200)]
Add instance state 'USER_down'

Add instance state 'USER_down' which is a state used in reporting only
and it represents the situation in which the user has shutdown the
instance but Ganeti's configuration still has this instance marked as
'ADMIN_up'.

Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

6 years agoPreserve Xen domain after shutdown
Jose A. Lopes [Tue, 22 Oct 2013 16:22:47 +0000 (18:22 +0200)]
Preserve Xen domain after shutdown

Change Xen domain configuration to preserve the domain state after
shutdown, as opposed to automatic reboot, which is the default
behaviour.

Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

6 years agoAdd Haskell hypervisor instance state
Jose A. Lopes [Wed, 23 Oct 2013 12:42:31 +0000 (14:42 +0200)]
Add Haskell hypervisor instance state

Add 'InstanceState' datatype which is the Haskell counterpart of the
Python type 'HvInstanceState'.

Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

6 years agoAdd Python hypervisor instance state
Jose A. Lopes [Tue, 22 Oct 2013 16:19:26 +0000 (18:19 +0200)]
Add Python hypervisor instance state

Add 'HvInstanceState' which represents the state of an instance
(either 'running' or 'shutdown') and fix the remaining hypervisor
backends to return a value of this type.  Before this patch, each
backend was returning their own value, some were returning strings,
others integers.  However, the value that represents the state of an
instance was never actually used in Ganeti, until instance shutdown
detection became necessary.

Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

6 years agoFix typo in documentation
Jose A. Lopes [Tue, 22 Oct 2013 16:18:12 +0000 (18:18 +0200)]
Fix typo in documentation

Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

6 years agoman gnt-network: indicate that --network is mandatory
Helga Velroyen [Wed, 13 Nov 2013 09:46:00 +0000 (10:46 +0100)]
man gnt-network: indicate that --network is mandatory

Despite the fact that the '--network' parameter of
'gnt-network add' is mandatory, it is listed in the man
page in parenthesis, which suggests that it is optional.
Fixing this to avoid user confusion.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

6 years agoAdd missing Constructor for SetParamsMods
Klaus Aehlig [Wed, 13 Nov 2013 09:47:33 +0000 (10:47 +0100)]
Add missing Constructor for SetParamsMods

Disks and nics can not only be addressed by indices, but also
by name. Hence add a constructor for this case as well, to be
faithful to the python world.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

6 years agoIgnore hlint warning "Error: Too strict if" in Server.hs
Petr Pudlak [Wed, 13 Nov 2013 12:21:09 +0000 (13:21 +0100)]
Ignore hlint warning "Error: Too strict if" in Server.hs

A previous patch [229da00] added an annotation for ignoring the warning,
but to the middle of a function, which doesn't compile. This patch moves
the annotation to the end of the function to correct the problem.

Signed-off-by: Petr Pudlak <pudlak@google.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoIgnore hlint warning "Error: Too strict if" in Server.hs
Petr Pudlak [Wed, 13 Nov 2013 09:36:23 +0000 (10:36 +0100)]
Ignore hlint warning "Error: Too strict if" in Server.hs

This warning appears only in newer versions of hlint (mine was v1.8.43),
and in this case it's reported incorrectly. The arguments to "showJSON"
have different types, therefore it's not possible to move "showJSON" in
front of "if".

Signed-off-by: Petr Pudlak <pudlak@google.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoDocument incompatible change of luxi
Klaus Aehlig [Mon, 11 Nov 2013 13:37:29 +0000 (14:37 +0100)]
Document incompatible change of luxi

With the change to luxid handling job queue requests, we
also inherited the parsing semantics of luxid, i.e., a
request is only considered syntactically correct, if all
parts are. Document that we decided to consider this change
of semantics desirable.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoAdapt TestJobSubmission to new semantics of SubmitManyJobs
Klaus Aehlig [Mon, 11 Nov 2013 12:15:02 +0000 (13:15 +0100)]
Adapt TestJobSubmission to new semantics of SubmitManyJobs

Historically, Ganeti would partially handle syntactically incorrect
luxi requests, if the outer structure was that of SubmitManyJobs
request. Moreover, this behavior was verified by a test. We changed to
completely reject syntactically incorrect luxi requests and decided
that this semantics was more desirable. Hence adapt the test
accordingly.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoUpdate future upgrade plans
Klaus Aehlig [Mon, 11 Nov 2013 15:52:41 +0000 (16:52 +0100)]
Update future upgrade plans

Now that mandatory opcode fields are mandatory also on disk,
document that, once we use soft drains during upgrades, the disk
representation of the job queue needs to be updated by cfgupgrade
as well.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoAdd a constructor to BaseOpCode
Klaus Aehlig [Mon, 11 Nov 2013 14:51:05 +0000 (15:51 +0100)]
Add a constructor to BaseOpCode

In order to guarantee all mandatory slots to be present, add
a custom constructor to BaseOpCode adding those fields with
default value, instead of inheriting the constructor from
outils.ValidatedSlots.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoFillDict: do not use exceptions for flow control
Santi Raffa [Mon, 11 Nov 2013 18:19:05 +0000 (18:19 +0000)]
FillDict: do not use exceptions for flow control

While it's generally better to beg for forgiveness than ask for
permission in Python, it's not a good idea to rely on exceptions to
model control flow, especially if the resulting code is simpler.

Signed-off-by: Santi Raffa <rsanti@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

6 years agoRemove hardcoded references to File, SharedFile templates
Santi Raffa [Mon, 11 Nov 2013 18:14:18 +0000 (18:14 +0000)]
Remove hardcoded references to File, SharedFile templates

DTS_FILEBASED is a constant that exists and this commit makes sure
that it is used whenever sensible, rather than resorting to hardcoding
the pair of templates in very many files.

Signed-off-by: Santi Raffa <rsanti@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>

6 years agoMake masterd call luxid to write jobs to queue
Klaus Aehlig [Wed, 6 Nov 2013 12:13:40 +0000 (13:13 +0100)]
Make masterd call luxid to write jobs to queue

As the responsibility for writing the job queue changed
to luxid, make masterd call to luxid when it is necessary
to write a job.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoSet reason trail for derived opcodes
Klaus Aehlig [Thu, 7 Nov 2013 11:44:34 +0000 (12:44 +0100)]
Set reason trail for derived opcodes

As the base parameters also includes the reason trail,
set it, if not set already, together with the other
base parameters in the case of an opcode expanding to
a list of opcodes.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoMake luxid handle SubmitManyJobs
Klaus Aehlig [Tue, 5 Nov 2013 16:37:28 +0000 (17:37 +0100)]
Make luxid handle SubmitManyJobs

Handle this request by writing the jobs to the
queue and inform masterd; masterd will then also
distribute the jobs to all master candidates.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoMake luxid handle SubmitJob
Klaus Aehlig [Fri, 11 Oct 2013 11:55:38 +0000 (13:55 +0200)]
Make luxid handle SubmitJob

As luxid is to take over responsibility for the job queue,
handle this request by writing the job to the queue and then
informing masterd; masterd will also distribute the job to
all master candidates.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoAdd the predicate of the queue being open
Klaus Aehlig [Wed, 6 Nov 2013 11:29:44 +0000 (12:29 +0100)]
Add the predicate of the queue being open

Adding jobs to the queue is only allowed if the queue
is not drained.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoProvide path to the queue drain file
Klaus Aehlig [Wed, 6 Nov 2013 10:52:15 +0000 (11:52 +0100)]
Provide path to the queue drain file

Since luxid is going to write to the job queue,
it needs to honor drains of the queue as well.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoAdd Luxi Request to pick up a job in the queue
Klaus Aehlig [Mon, 28 Oct 2013 16:17:27 +0000 (17:17 +0100)]
Add Luxi Request to pick up a job in the queue

During the transition to the new daemon layout, from step 2
onwards, luxid will write to the queue but masterd will trigger
the execution. Therefore, add a new luxi request to tell masterd
to pick up a job that has already been written to the queue.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoFactor out functionality to pickup a job from the queue
Klaus Aehlig [Mon, 28 Oct 2013 14:01:41 +0000 (15:01 +0100)]
Factor out functionality to pickup a job from the queue

The current restart procedure for masterd includes functionality
to pick up a job from the queue and restart it, if it hasn't been
started before. Move this functionality into a separate function
to be able to have the enqueuing be done by luxid.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoProvide means of locking a file
Klaus Aehlig [Wed, 23 Oct 2013 08:55:13 +0000 (10:55 +0200)]
Provide means of locking a file

Two avoid two processes simultaneously accessing the same
on-file structure, like the job queue, file locks are used.
Therefore, provide this functionality in Haskell as well.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoProvide path to the queue lock file
Klaus Aehlig [Tue, 22 Oct 2013 15:44:22 +0000 (17:44 +0200)]
Provide path to the queue lock file

To avoid several processes accessing the queue at the same time,
Ganeti locks the queue via a lock file on disk. Provide the path
to this file.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoProvide a function to write jobs to disk
Klaus Aehlig [Wed, 16 Oct 2013 14:27:22 +0000 (16:27 +0200)]
Provide a function to write jobs to disk

This function writes a (non-archived) job to disk. The file
name can be computed from the job id, which is part of the job.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoProvide means to allocate new job ids
Klaus Aehlig [Wed, 16 Oct 2013 12:20:54 +0000 (14:20 +0200)]
Provide means to allocate new job ids

Add utility functions to allocate new job ids by increasing
the value stored in the serial file. As this function is
used in a multi-threaded program, synchronize access over
an internal lock.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoAdd function to get master candidates from configuration
Klaus Aehlig [Tue, 5 Nov 2013 13:18:18 +0000 (14:18 +0100)]
Add function to get master candidates from configuration

With luxi daemon taking over part of the job queue management, it will
also be responsible for replicating the queue to all master candidates.
Therefore, add a function to extract the list of master candidates from
the configuration.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoSupport RPC asking to replicate part of the job queue
Klaus Aehlig [Mon, 4 Nov 2013 17:26:28 +0000 (18:26 +0100)]
Support RPC asking to replicate part of the job queue

To be able to replicate the job queue, in particular the serial,
luxid needs to be able to send the jobqueue_update RPC. So add
its definition.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoProvide function to obtain the unique element of a list
Klaus Aehlig [Wed, 16 Oct 2013 11:45:11 +0000 (13:45 +0200)]
Provide function to obtain the unique element of a list

This version of 'the' properly lives in the 'Result' monad,
as opposed to traditional one calling 'error'. The reason
why it is 'Bad' that not precisely one element is returned
is given as argument.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoProvide method to read job serial number
Klaus Aehlig [Tue, 15 Oct 2013 16:35:03 +0000 (18:35 +0200)]
Provide method to read job serial number

This methods allows reading the maximal job serial number from
disk.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoProvide convenience function to create Job from op-codes
Klaus Aehlig [Tue, 15 Oct 2013 11:17:45 +0000 (13:17 +0200)]
Provide convenience function to create Job from op-codes

This function handles the pure part of generating a job,
i.e., assuming the job id already assigned and not setting
time stamps.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoAdd function to resolve dependencies in meta op code
Klaus Aehlig [Wed, 6 Nov 2013 15:48:38 +0000 (16:48 +0100)]
Add function to resolve dependencies in meta op code

When queueing many jobs, the dependencies between them need to
be resolved with the knowledge of their respective job id.
Lift the computation of the absolute dependency to the level
of MetaOpCodes.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoAdd function to compute the absolute id of a dependency
Klaus Aehlig [Wed, 6 Nov 2013 15:01:26 +0000 (16:01 +0100)]
Add function to compute the absolute id of a dependency

SubmitManyJobs also accepts jobs with dependencies given as
relative ids. Together with the absolute id of the job, once
assigned, the dependency can be resolved. Add a function doing
this computation.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoProvide a convenience method to optain a QueuedOpCode
Klaus Aehlig [Mon, 14 Oct 2013 16:09:49 +0000 (18:09 +0200)]
Provide a convenience method to optain a QueuedOpCode

When generating jobs from sequences of op-codes, it is necessary
to wrap op-codes into queued form.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoAdd utility function tryAndLogIOError
Klaus Aehlig [Wed, 16 Oct 2013 15:32:18 +0000 (17:32 +0200)]
Add utility function tryAndLogIOError

This function allows to use 'IO a' objects in a safe
way, using the 'try' function; the outcome is reported
as a 'Result'. IOErrors are logged and the result is
'Bad', while in the case of no errors, a result-yielding
function is applied to the value.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoProvide utility to atomically write a file
Klaus Aehlig [Wed, 16 Oct 2013 08:57:51 +0000 (10:57 +0200)]
Provide utility to atomically write a file

To keep our on-file data consistent at any moment, we change
file contents by atomically replacing the file with a new one.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoDesign document for custom SSH ports
Petr Pudlak [Fri, 8 Nov 2013 16:21:38 +0000 (17:21 +0100)]
Design document for custom SSH ports

See issue 235. It turned out the feature is a bit more complex than expected,
hence this design doc.

Signed-off-by: Petr Pudlak <pudlak@google.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoOptimize Haskell compilation
Jose A. Lopes [Wed, 18 Sep 2013 11:22:02 +0000 (13:22 +0200)]
Optimize Haskell compilation

The idea is to compile (on demand, that is, when necessary) each
source file to a normal object file, a coverage object file, a
profiling object file, and a test object file.  Also, a given Haskell
binary is linked with the proper object files.  This is achieved with
the following Makefile variables:

Compilation modes (profiles):
1. HPROFILE enables/disables profiling
2. HCOVERAGE enables/disables coverage
3. HTEST enables/disables 'TEST' preprocessor definition

A few words on testing: testing means the problem described in issue
535: https://code.google.com/p/ganeti/issues/detail?id=535.  With
HTEST enable, ghc will be instructed to define the preprocessor
definition TEST, for modules that use '#ifdef TEST'.

Haskell binary targets fetch the proper dependencies.  They are also
'.PHONY' targets so that 'make' does not check for the file timestamp
and instead it will always call 'ghc --make ...'.  This is not a
problem because 'ghc' with the '--make' flag only compiles the
necessary object files.

Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoFix getNodeRole
Klaus Aehlig [Tue, 5 Nov 2013 14:11:32 +0000 (15:11 +0100)]
Fix getNodeRole

In the configuration, the master node is now given
by its uuid. Therefore, compare the uuid and not the
name to find out if a given node is the master.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoMerge branch 'stable-2.10' into master
Klaus Aehlig [Tue, 5 Nov 2013 13:00:36 +0000 (14:00 +0100)]
Merge branch 'stable-2.10' into master

* stable-2.10
  Fix documentation
  Replace all constant definitions with re-exports
  Prepare constants for automatic reexport
  Hs2Py constants: 'hvsParameterTypes' and 'hvsParameters'
  Fix indentation that triggers PEP8 error
  Check if hotplug is supported in CheckPrereq

* stable-2.9
  Revision bump for 2.9.0
  Schedule release of 2.9.0

* stable-2.8
  Improve error message for replace-disks

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoMerge branch 'stable-2.9' into stable-2.10
Klaus Aehlig [Tue, 5 Nov 2013 12:47:30 +0000 (13:47 +0100)]
Merge branch 'stable-2.9' into stable-2.10

* stable-2.9
  Revision bump for 2.9.0
  Schedule release of 2.9.0

* stable-2.8
  Improve error message for replace-disks

Conflicts:
NEWS: trivial
configure.ac: drop suffix bump of stable-2.9

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoRevision bump for 2.9.0 v2.9.0
Klaus Aehlig [Mon, 4 Nov 2013 09:40:13 +0000 (10:40 +0100)]
Revision bump for 2.9.0

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoSchedule release of 2.9.0
Klaus Aehlig [Mon, 4 Nov 2013 09:39:36 +0000 (10:39 +0100)]
Schedule release of 2.9.0

...and mention the last change pulled in from stable-2.8.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoMerge branch 'stable-2.8' into stable-2.9
Klaus Aehlig [Mon, 4 Nov 2013 15:36:02 +0000 (16:36 +0100)]
Merge branch 'stable-2.8' into stable-2.9

* stable-2.8
  Improve error message for replace-disks

Conflicts:
lib/cmdlib/instance_storage.py
Resolved by manually applying the node name to uuid
transition on the version of stable-2.9.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoImprove error message for replace-disks
Michele Tartara [Mon, 4 Nov 2013 15:20:07 +0000 (15:20 +0000)]
Improve error message for replace-disks

In some conditions, replace-disks will fail if the disks are not properly
activated. Improve the error message suggesting to run activate-disks before
executing replace-disks.

Fixes Issue 606.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoFix documentation
Jose A. Lopes [Wed, 30 Oct 2013 09:25:42 +0000 (10:25 +0100)]
Fix documentation

Fix documentation in constants containing values in seconds.

Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoReplace all constant definitions with re-exports
Jose A. Lopes [Wed, 30 Oct 2013 09:15:57 +0000 (10:15 +0100)]
Replace all constant definitions with re-exports

Remove all constant definitions in 'lib/constants.py' and re-export
the module 'lib/_constants.py', which contains the generated
constants.

Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoPrepare constants for automatic reexport
Jose A. Lopes [Mon, 28 Oct 2013 18:01:48 +0000 (19:01 +0100)]
Prepare constants for automatic reexport

* separate constants and non-constants in 'lib/constants.py' to make
  it simple to replace their re-export definitions with a single
  re-export statement at the begining of the module

* fix access 'CONFD_REQFIELD_*' related constants in
  'tools/confd-client' to convert them to 'String', rather then doing
  it in 'lib/constants.py' in order to allow the replacement of the
  re-export definitions with a single re-export statement

Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoHs2Py constants: 'hvsParameterTypes' and 'hvsParameters'
Jose A. Lopes [Mon, 28 Oct 2013 13:45:40 +0000 (14:45 +0100)]
Hs2Py constants: 'hvsParameterTypes' and 'hvsParameters'

Add constants 'hvsParameters' and 'hvsParameterTypes' to the Haskell
to Python constant generation.

Signed-off-by: Jose A. Lopes <jabolopes@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoAdd visual/color marks to the QA script
Petr Pudlak [Mon, 4 Nov 2013 07:44:14 +0000 (08:44 +0100)]
Add visual/color marks to the QA script

 * Use different visual marks at the beginning of lines (such as "<<<<" for
    starting a test and ">>>>" for finishing it)

 * Show more specifically if a test fails/passes.

 * If stdout is a terminal, use colors to distinguish different messages

Signed-off-by: Petr Pudlak <pudlak@google.com>
Reviewed-by: Jose A. Lopes <jabolopes@google.com>

6 years agoAdd the function name to textual descriptions of tests
Petr Pudlak [Thu, 31 Oct 2013 14:36:30 +0000 (15:36 +0100)]
Add the function name to textual descriptions of tests

It makes it a bit easier to find the corresponding functions.

Signed-off-by: Petr Pudlak <pudlak@google.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoFix indentation that triggers PEP8 error
Petr Pudlak [Thu, 31 Oct 2013 14:35:00 +0000 (15:35 +0100)]
Fix indentation that triggers PEP8 error

Signed-off-by: Petr Pudlak <pudlak@google.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agohsqueeze: document expansion planning in man page
Klaus Aehlig [Thu, 31 Oct 2013 13:53:22 +0000 (14:53 +0100)]
hsqueeze: document expansion planning in man page

Describe the new behavior of hsqueeze to also plan taking nodes online
in the man page.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

6 years agoAdd a test for hsqueeze expansion
Klaus Aehlig [Wed, 30 Oct 2013 16:45:11 +0000 (17:45 +0100)]
Add a test for hsqueeze expansion

In this example, there are 7 nodes, each with capacity for
4 instances. Two nodes are online and host 4 instances each,
thus forming a cluster with no free resources. Four additional
nodes a standby, and two additional node is offline for other
reasons.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

6 years agohsqueeze: support planing for onlining nodes
Klaus Aehlig [Thu, 31 Oct 2013 09:42:29 +0000 (10:42 +0100)]
hsqueeze: support planing for onlining nodes

If the amount of free resources falls below a given threshold,
hsqueeze will suggest putting standby nodes back online until
the minimum of free resources is reached, or all standby nodes
are online.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

6 years agoAdd an --minimal-resources option
Klaus Aehlig [Tue, 29 Oct 2013 15:46:29 +0000 (16:46 +0100)]
Add an --minimal-resources option

Add a new option, to be used by hsqueeze, to specify
the amount of free resources that has to be on each
node, in order not to start onlining standby nodes.
It is given as a multiple of the standard allocation,
as specified by the instance policy.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

6 years agoText Backend: correctly read data for offline nodes
Klaus Aehlig [Wed, 30 Oct 2013 17:41:32 +0000 (18:41 +0100)]
Text Backend: correctly read data for offline nodes

With standby nodes, simply ignoring the specification of
an offline node is not sufficient any more.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

6 years agohsqueeze: add a test for the -v option
Klaus Aehlig [Wed, 30 Oct 2013 13:54:16 +0000 (14:54 +0100)]
hsqueeze: add a test for the -v option

This test also verifies that the master node is
not considered for being offlined.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

6 years agoChange default for target resource to 2.0
Klaus Aehlig [Tue, 29 Oct 2013 13:36:04 +0000 (14:36 +0100)]
Change default for target resource to 2.0

The target resources, as used by hsqueeze, are supposed to
be strictly higher than the minimal resources. However,
keeping minimal resources of less than a single instance
is not a useful reserve.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

6 years agogit-ignore test/hs/hsqueeze
Klaus Aehlig [Wed, 30 Oct 2013 15:53:22 +0000 (16:53 +0100)]
git-ignore test/hs/hsqueeze

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

6 years agoconfigure: avoid duplicate Haskell module checking
Klaus Aehlig [Thu, 31 Oct 2013 12:58:59 +0000 (13:58 +0100)]
configure: avoid duplicate Haskell module checking

There are some Haskell modules, that are optional for production,
but needed in test. Check for those only once, by unconditionally
checking before considering the modules for the specific feature.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

6 years agoMerge branch 'stable-2.10' into master
Klaus Aehlig [Wed, 30 Oct 2013 16:21:29 +0000 (17:21 +0100)]
Merge branch 'stable-2.10' into master

* stable-2.10
  Remove FIXME + update man pages wrt iallocator
  Don't check for master netdev when using OVS
  Fix TypeError in backend/ConfigureOVS
  Create Open vSwitch on Master during Cluster init
  SimpleRetry on BlockDev.Remove()
  Show the key in "'None' is not a valid Maybe value"
  Fix usage of xm/xl for gnt-instance modify --new-primary
  Hs2Py constants (17 patches)
  Fix default value for reason trail
  Hotplug (23 patches)

* stable-2.8
  Add all dependencies for confd as test dependencies
  Add snap-server to the test-relevant packages
  Placate warnings on ganeti.outils_unittest.py

Conflicts:
lib/cmdlib/instance.py: trivial
src/Ganeti/Types.hs: use both additions
test/py/cmdlib/instance_unittest.py: manually
            appply ff5def9 to version of master

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoCheck if hotplug is supported in CheckPrereq
Dimitris Aragiorgis [Wed, 30 Oct 2013 16:56:01 +0000 (18:56 +0200)]
Check if hotplug is supported in CheckPrereq

Introduce new RPC hotplug_supported that invokes the corresponding
hypervisor's method which checks if hotplug is generally
supported. Call this RPC early in CheckPrereq() and abort if
hotplug is not supported. Currently only KVM hypervisor with
qemu versions > 1.0. Please note that this check is unaware of
target device and hotplug action. This latter check is still
done during hotplug_device() RPC.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoMerge branch 'stable-2.9' into stable-2.10
Klaus Aehlig [Wed, 30 Oct 2013 13:07:52 +0000 (14:07 +0100)]
Merge branch 'stable-2.9' into stable-2.10

* stable-2.9
  (no changes)

* stable-2.8
  Add all dependencies for confd as test dependencies
  Add snap-server to the test-relevenat packages
  Placate warnings on ganeti.outils_unittest.py

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

6 years agoMerge branch 'stable-2.8' into stable-2.9
Klaus Aehlig [Wed, 30 Oct 2013 12:33:58 +0000 (13:33 +0100)]
Merge branch 'stable-2.8' into stable-2.9

* stable-2.8
  Add all dependencies for confd as test dependencies
  Add snap-server to the test-relevenat packages
  Placate warnings on ganeti.outils_unittest.py

Conflicts:
configure.ac: take both additions (and fix)

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoRemove FIXME + update man pages wrt iallocator
Helga Velroyen [Fri, 25 Oct 2013 15:14:17 +0000 (17:14 +0200)]
Remove FIXME + update man pages wrt iallocator

This patch removes a FIXME regarding the retrieval
of the disk template for computation of the cluster
information for the iallocator. After examining the
code, I figured out that it is actually a quite
robust way to just check if the disk template is
available, in particular with respect to extensions
of the iallocator.

In this context, I also updated the man pages of
the operations that use the iallocator, because so
far it was not well documented which disk template
is considered when more than one instance was
allocated. The situation is not ideal (the default
disk template is used), but it is still better than
in 2.9. where only LVM storage was considered.
With the implementation of storage pools, it would
make sense to extend the iallocator protocol to
include storage information for more than one
template.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoAdd all dependencies for confd as test dependencies
Klaus Aehlig [Wed, 30 Oct 2013 10:13:48 +0000 (11:13 +0100)]
Add all dependencies for confd as test dependencies

Since our tests pull in confd as a dependency, all build dependencies
for confd are also necessary to run the tests.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoDon't check for master netdev when using OVS
Sebastian Gebhard [Mon, 14 Oct 2013 13:50:42 +0000 (13:50 +0000)]
Don't check for master netdev when using OVS

This patch disables checking if the given master_netdev exists during cluster
init.
When OVS is used, the needed Open vSwitch will be created later on (with
OpClusterPostInit) and is thus not available at this point of time.

Signed-off-by: Sebastian Gebhard <sege@fs.ei.tum.de>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoFix TypeError in backend/ConfigureOVS
Sebastian Gebhard [Mon, 14 Oct 2013 13:54:04 +0000 (13:54 +0000)]
Fix TypeError in backend/ConfigureOVS

In case of running into a failure during Open vSwitch creation, the error is
not given, but a TypeError is thrown. This is because the number of arguments
for the format string was not equal to the place holders.

This fixes the issue.

Signed-off-by: Sebastian Gebhard <sege@fs.ei.tum.de>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoCreate Open vSwitch on Master during Cluster init
Sebastian Gebhard [Mon, 14 Oct 2013 13:54:04 +0000 (13:54 +0000)]
Create Open vSwitch on Master during Cluster init

Currently, Open vSwitch is only created on additional nodes, not on the master
itself.
This fixes the issue and creates and configures an Open vSwitch on the master
node during cluster init.

Signed-off-by: Sebastian Gebhard <sege@fs.ei.tum.de>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoAdd design document for multi-version tests
Hrvoje Ribicic [Mon, 28 Oct 2013 14:30:11 +0000 (14:30 +0000)]
Add design document for multi-version tests

This patch adds a design document for multi-version tests, discussing
what they aim to do and how they should be implemented.

Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoSimpleRetry on BlockDev.Remove()
Dimitris Aragiorgis [Fri, 25 Oct 2013 15:43:06 +0000 (18:43 +0300)]
SimpleRetry on BlockDev.Remove()

Sometimes, upon disk removal, corresponding file descriptors
are kept briefly open by various processes (hypervisor, blkid, etc.).
With this patch, we retry several times before raising the appropriate
error, thus making disk removal more robust against those corner cases.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoAdd snap-server to the test-relevenat packages
Klaus Aehlig [Tue, 29 Oct 2013 15:09:14 +0000 (16:09 +0100)]
Add snap-server to the test-relevenat packages

While snap-server is only needed for the optional monitoring daemon,
some tests, notably those testing these optional features, still depend
on it. So, if snap-server is missing, the Haskell tests should not be
run, as they cannot even be build.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoMention compression for moves/backups in NEWS
Thomas Thrainer [Fri, 18 Oct 2013 21:15:23 +0000 (23:15 +0200)]
Mention compression for moves/backups in NEWS

Mention the possibility to specify compression for transfers / instance
storage in NEWS file.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoAdd --compress option to instance_move
Thomas Thrainer [Fri, 18 Oct 2013 21:10:01 +0000 (23:10 +0200)]
Add --compress option to instance_move

instance_move now supports the --compress option with which it's
possible to select the compression mode to use during the instance move.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoAdd possibility to compress to OpInstanceCreate
Thomas Thrainer [Fri, 18 Oct 2013 20:39:45 +0000 (22:39 +0200)]
Add possibility to compress to OpInstanceCreate

OpInstanceCreate now supports the 'compress' option. It allows to enable
compression during instance imports.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoHonor compress parameter in remote backups
Thomas Thrainer [Fri, 18 Oct 2013 20:31:16 +0000 (22:31 +0200)]
Honor compress parameter in remote backups

Use the specified compression mode not only for local exports, but also
for remote ones.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoAdd --compress option to gnt-backup import
Thomas Thrainer [Thu, 24 Oct 2013 15:00:29 +0000 (17:00 +0200)]
Add --compress option to gnt-backup import

Add a new command line parameter to gnt-backup import and pass
its value to OpInstanceCreate. Also document the option in the
man page.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoAdd --compress option to gnt-backup export
Thomas Thrainer [Fri, 18 Oct 2013 20:16:49 +0000 (22:16 +0200)]
Add --compress option to gnt-backup export

Add a new command line parameter to gnt-backup export and pass its value
to OpBackupExport. Also document the option in the man page.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoSupport import with compressed instance moves
Thomas Thrainer [Fri, 18 Oct 2013 20:09:29 +0000 (22:09 +0200)]
Support import with compressed instance moves

Support compressing instance data while sending it to the target node on
instance imports.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoAdd local compression to OpBackupExport
Thomas Thrainer [Fri, 18 Oct 2013 19:54:30 +0000 (21:54 +0200)]
Add local compression to OpBackupExport

OpBackupExport is extended by a compress parameter. This parameter
(either 'none' or 'gzip') controls if instance disks are compressed
before being sent over the network to the destination node.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoAdd --compress option to gnt-instance move
Thomas Thrainer [Fri, 18 Oct 2013 19:25:03 +0000 (21:25 +0200)]
Add --compress option to gnt-instance move

Add a new command line parameter to gnt-instance and pass its value to
OpInstanceMove. Also document the option in the man page.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoAdd possibility to compress to OpInstanceMove
Thomas Thrainer [Fri, 18 Oct 2013 18:27:15 +0000 (20:27 +0200)]
Add possibility to compress to OpInstanceMove

OpInstanceMove now supports the 'compress' option. It allows to enable
compression for intra-cluster instance moves.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoRemove trailing whitespace
Thomas Thrainer [Fri, 18 Oct 2013 18:31:39 +0000 (20:31 +0200)]
Remove trailing whitespace

Remove trailing whitespace from OpCodes.hs and OpParams.hs.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoRemove blockdev_export RPC
Thomas Thrainer [Wed, 9 Oct 2013 14:47:52 +0000 (16:47 +0200)]
Remove blockdev_export RPC

This RPC call is no longer used, as for intra-cluster instance moves the
import/export daemon is used too.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoOptimize dd parameters for instances moves
Thomas Thrainer [Wed, 9 Oct 2013 14:42:06 +0000 (16:42 +0200)]
Optimize dd parameters for instances moves

Benchmarks have shown that increasing the buffer size and disabling
synchronous output increased the throughput by about 10MiB/s, so make
those parameters the default.

Note that not using sync for output doesn't increase the risk of data
loss, as the data will remain on the source node anyway if an error
occurs during the move. This change, and increasing the block size to 1
MiB, didn't increase the memory consumption of dd during the move
operation significantly.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoUse import-export daemon for intra-cluster moves
Thomas Thrainer [Wed, 9 Oct 2013 11:40:00 +0000 (13:40 +0200)]
Use import-export daemon for intra-cluster moves

This unifies the inter- and intra-cluster moves and the backup code, so
less code needs to be maintained.

Also fix failing tests for LUInstanceMove by following the code changes
in the tests.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoUse cluster variable consistently
Thomas Thrainer [Wed, 9 Oct 2013 08:57:24 +0000 (10:57 +0200)]
Use cluster variable consistently

The cluster object is already retrieved from the configuration, so use
it consistently in LUInstanceMove.CheckPrereq().

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoCorrect comments
Thomas Thrainer [Wed, 9 Oct 2013 08:51:16 +0000 (10:51 +0200)]
Correct comments

Correct comments which confuse the target node with the secondary node
of an instance in LUInstanceMove.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoFix expectation of the htools wrapping
Klaus Aehlig [Tue, 29 Oct 2013 13:08:13 +0000 (14:08 +0100)]
Fix expectation of the htools wrapping

With the addition of hsqueeze, htools obtained a new personality.
Additionally, the longer htool name also changes the line wrapping of
the descriptions. Adapt the expectation accordingly.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>

6 years agoAdd a man page for hsqueeze
Klaus Aehlig [Fri, 25 Oct 2013 15:37:12 +0000 (17:37 +0200)]
Add a man page for hsqueeze

Describe the current state of the Ganeti dynamic power management
tool, hsqueeze, in a man page.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoAdd test for hsqueeze compression
Klaus Aehlig [Thu, 24 Oct 2013 11:47:44 +0000 (13:47 +0200)]
Add test for hsqueeze compression

In this example cluster, there are 6 nodes with
capacity for 4 instances each, holding a total
of 8 instances. So even after turning off 2 nodes
and rebalancing, every node still has capacity for
2 instances.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoAdd hsqueeze planning for compression
Klaus Aehlig [Thu, 24 Oct 2013 09:24:07 +0000 (11:24 +0200)]
Add hsqueeze planning for compression

Add a new htool, hsqueeze, for dynamic power management. This
commit only implements the first useful part: plan (but not
executing) taking nodes offline nodes still keeping within the
resource limit.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoProvide --target-resources option
Klaus Aehlig [Fri, 25 Oct 2013 11:52:24 +0000 (13:52 +0200)]
Provide --target-resources option

Add a new option, to be used by hsqueeze, to specify the
target free resources on each node. It is given as a multiple
of the standard allocation, as specified in the instance policy.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

6 years agoShow the key in "'None' is not a valid Maybe value"
Petr Pudlak [Mon, 28 Oct 2013 14:53:28 +0000 (15:53 +0100)]
Show the key in "'None' is not a valid Maybe value"

Currently the error message doesn't say which key is affected, which
makes it kind of useless.

Signed-off-by: Petr Pudlak <pudlak@google.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoAdd RpcResult.__repr__ for debugging
Petr Pudlak [Mon, 28 Oct 2013 14:52:13 +0000 (15:52 +0100)]
Add RpcResult.__repr__ for debugging

This way it's possible to easily log the whole result in a human
readable form.

Signed-off-by: Petr Pudlak <pudlak@google.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoLet git ignore `.hsenv` directory and vim's session file
Petr Pudlak [Mon, 28 Oct 2013 14:02:11 +0000 (15:02 +0100)]
Let git ignore `.hsenv` directory and vim's session file

Signed-off-by: Petr Pudlak <pudlak@google.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>

6 years agoIf a configuration file can't be loaded, show which one
Petr Pudlak [Mon, 28 Oct 2013 13:59:44 +0000 (14:59 +0100)]
If a configuration file can't be loaded, show which one

So far the error message only says it can't be loaded, but not what file
it is.

Signed-off-by: Petr Pudlak <pudlak@google.com>
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>