ganeti-github.git
9 years agoMerge branch 'devel-2.4'
Michael Hanselmann [Mon, 25 Jul 2011 13:02:23 +0000 (15:02 +0200)]
Merge branch 'devel-2.4'

* devel-2.4:
  Reopen daemon's stdio on SIGHUP
  Reopen log file only once after SIGHUP
  Don't leak file descriptors when setting up daemon output
  Fix aliases in bash completion

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoReopen daemon's stdio on SIGHUP
Michael Hanselmann [Mon, 25 Jul 2011 12:35:51 +0000 (14:35 +0200)]
Reopen daemon's stdio on SIGHUP

Before this patch daemons would continue to refer to an old logfile for
their standard I/O if they had been asked to reopen the log (SIGHUP).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoReopen log file only once after SIGHUP
Michael Hanselmann [Mon, 25 Jul 2011 11:08:22 +0000 (13:08 +0200)]
Reopen log file only once after SIGHUP

Commit b6fa9a44 added a re-openable log handler. The log file is
reopened when a daemon is sent a HUP signal. Due to a bug in the code,
fixed by this patch, the log file would be reopened for every single log
message thereafter.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoDon't leak file descriptors when setting up daemon output
Michael Hanselmann [Mon, 25 Jul 2011 10:02:24 +0000 (12:02 +0200)]
Don't leak file descriptors when setting up daemon output

When a daemon's output is configured using “utils.SetupDaemonFDs”, the
function must use dup2(2). Unfortunately the code didn't close the
original file descriptors, leaking them in the process.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agohtools: rework the algorithm for ChangeAll mode
Iustin Pop [Mon, 18 Jul 2011 11:17:41 +0000 (13:17 +0200)]
htools: rework the algorithm for ChangeAll mode

I think I've identified the problem with the current ChangeAll
mode. The current algorithm works as follows:

- identify a new primary by choosing the node which gives best score
  as new secondary
- failover to it
- identify a new secondary by choosing the node which gives best score
  as new secondary

This means that the future primary is 'fixed' after the first
iteration, leaving to possibly suboptimal results. This patch changes
the algorithm to do what, in hindsight, seems the obvious thing to do:
- generate all pairs (primary, secondary)
- identify the pair that after the above sequence (r:np, f, r:ns)
  gives the best group score

This fixes some of the corner cases I've seen in relocation, but not
all; the remaining cases are related to multi-instance relocation and
while they can't be fixed in the current framework, the needed
rebalancing is much smaller than with the current algorithm.

The patch also fixes an issue with the docstring of another function.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

9 years agognt-instance info: Return static info if node offline
Michael Hanselmann [Fri, 22 Jul 2011 11:27:05 +0000 (13:27 +0200)]
gnt-instance info: Return static info if node offline

Before this patch “gnt-instance info” would fail with the error message
“Error checking node $node: Node is marked offline” if the instance's
primary node is marked offline and the user didn't explicitely request
static information only. With this patch the LU will automatically
return static information if the instance's primary node is marked
offline.

Some explicit loops are changed to map().

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoIgnore offline primary when failing over
Michael Hanselmann [Fri, 22 Jul 2011 11:04:57 +0000 (13:04 +0200)]
Ignore offline primary when failing over

When the source node for a failover is marked offline, there's no need
to require the user to specify “--ignore-consistency”.

To make it work at all, a number of bugs introduced by the merge of
migration and failover are also fixed by this patch.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agohtools: replace two hardcoded uses of pri+sec nodes
Iustin Pop [Fri, 15 Jul 2011 08:53:45 +0000 (10:53 +0200)]
htools: replace two hardcoded uses of pri+sec nodes

These two cases use explicit uses of primary and secondary nodes with
Instance.allNodes, which means the code is more flexible if the
internal layout of the instance changes.

I've verified that the output of involvedNodes  is not required to be
4-element long, and as such the function docstring has been updated.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: add target_node member to migrate opcode
Iustin Pop [Sat, 9 Jul 2011 17:48:36 +0000 (19:48 +0200)]
htools: add target_node member to migrate opcode

… and failover too. Not many changes otherwise except for
serialisation and unittests.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: do not change node disk for non-local storage
Iustin Pop [Sat, 9 Jul 2011 09:17:10 +0000 (11:17 +0200)]
htools: do not change node disk for non-local storage

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: add more functions for local disk storage
Iustin Pop [Sat, 9 Jul 2011 09:01:49 +0000 (11:01 +0200)]
htools: add more functions for local disk storage

These will be used in Node.hs for proper add/remove instance
code. Furthermore, we restrict the movable status to the right disk
templates only, so that we don't attempt to move the 'wrong' instance
types.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agoInitial design doc for OVF support
Agata Murawska [Thu, 21 Jul 2011 15:31:44 +0000 (17:31 +0200)]
Initial design doc for OVF support

Signed-off-by: Agata Murawska <agatamurawska@google.com>
[iustin@google.com: fixed formatting issues]

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

9 years agoFix aliases in bash completion
Michael Hanselmann [Fri, 22 Jul 2011 09:55:46 +0000 (11:55 +0200)]
Fix aliases in bash completion

Ever since commit 2d48a3a2 aliases were not included in the bash
completion script. This patch also replaces one tab with two spaces.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agognt-instance console: Use query instead of opcode
Michael Hanselmann [Fri, 22 Jul 2011 06:14:39 +0000 (08:14 +0200)]
gnt-instance console: Use query instead of opcode

This means opening the console no longer requires the instance lock,
allowing it to be used during long-running operations (e.g. replacing a
disk).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoMerge branch 'devel-2.4'
Michael Hanselmann [Fri, 22 Jul 2011 09:05:55 +0000 (11:05 +0200)]
Merge branch 'devel-2.4'

* devel-2.4:
  gnt-node volumes: Fix instance names

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoAdd opcode attribute for comments
Michael Hanselmann [Fri, 22 Jul 2011 05:35:44 +0000 (07:35 +0200)]
Add opcode attribute for comments

This attribute allows programmatic submitters of jobs (e.g. iallocator)
to add a comment to each opcode, describing its purpose. Example:

$ gnt-job info 123
Job ID: 123
  …
  Opcodes:
    OP_INSTANCE_REPLACE_DISKS
      …
      Input fields:
        comment: Replaces disks on inst1.example.com
      …

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agognt-node volumes: Fix instance names
Michael Hanselmann [Fri, 22 Jul 2011 08:27:09 +0000 (10:27 +0200)]
gnt-node volumes: Fix instance names

Commit 84d7e26b changed “objects.Instance.MapLVsByN” to not just return
the LV name, but to include the volume group name (e.g.
“xenvg/d67e8700….disk0_data”). This in turn broke the mapping of volume
names in LUNodeQueryvols, stopping instance names from displayed in
“gnt-node volumes”.

This patch fixes the issue and does some cleanup.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoFixed one option name and a typo in the docs
Andrea Spadaccini [Thu, 21 Jul 2011 15:54:42 +0000 (16:54 +0100)]
Fixed one option name and a typo in the docs

The -g vg-name option was deprecated in commit
04367e70ad71eea3f0f19e7889dc68fb9783c98a.

Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoFix instance failover (missing argument)
Michael Hanselmann [Thu, 21 Jul 2011 13:22:23 +0000 (15:22 +0200)]
Fix instance failover (missing argument)

More fallout from commit 323f9095b49d.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoImplement instance failover via RAPI
Michael Hanselmann [Thu, 21 Jul 2011 13:20:45 +0000 (15:20 +0200)]
Implement instance failover via RAPI

No idea why this was missed before.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoExport job dependencies through lock monitor
Michael Hanselmann [Thu, 21 Jul 2011 08:47:27 +0000 (10:47 +0200)]
Export job dependencies through lock monitor

This makes them visible to the user. Example:

$ gnt-debug locks -o name,pending
Name    Pending
job/890 job:891,892
job/892 job:894

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agolocking.GLM: Allow adding locks to monitor
Michael Hanselmann [Thu, 21 Jul 2011 08:49:21 +0000 (10:49 +0200)]
locking.GLM: Allow adding locks to monitor

This will be used for exporting job dependencies through
the lock monitor.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoMake lock monitor more versatile
Michael Hanselmann [Wed, 13 Jul 2011 20:43:22 +0000 (22:43 +0200)]
Make lock monitor more versatile

With this change it'll be possible to register other lock information
providers. One usecase for this are job dependencies, which can be shown
in the output of “gnt-debug locks”, too.

The lock monitor is changed to accept more than one return value from
the function providing the information. Unfortunately it's hard to keep
weak references to bound methods, so that I settled on keeping a weak
reference on the object instead (see note in docstring).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoUpdate documentation regarding Haskell dependencies
Iustin Pop [Fri, 8 Jul 2011 14:07:42 +0000 (16:07 +0200)]
Update documentation regarding Haskell dependencies

These were forgot when the supported library versions were changed.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

9 years agohtools: add two more small unittests
Iustin Pop [Fri, 8 Jul 2011 13:52:14 +0000 (15:52 +0200)]
htools: add two more small unittests

This adds tests for the opToResult and eitherToResult functions from
Types.hs, and changes two other tests for the same module to test JSON
serialisation (which automatically also tests the lower-level to/from
string conversion functions).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

9 years agohtools: update hail man page with the new modes
Iustin Pop [Fri, 8 Jul 2011 13:23:26 +0000 (15:23 +0200)]
htools: update hail man page with the new modes

Also mark the deprecated modes we no longer support.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

9 years agohtools: a few more hlint fixes
Iustin Pop [Fri, 8 Jul 2011 13:18:07 +0000 (15:18 +0200)]
htools: a few more hlint fixes

Tested only on GHC 7.x, will test on 6.1x too before commit.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

9 years agohtools: further docstring fixes
Iustin Pop [Fri, 8 Jul 2011 12:52:01 +0000 (14:52 +0200)]
htools: further docstring fixes

This adds parameter documentation for Cluster.iMoveToJob (I think it
was not clear if the new or old node list is needed) and fixes other
docstring style issues.

After this patch, all modules except for CLI.hs (which has many
obvious declarations for command-line options) and QC.hs (unittests)
have 100% doc-strings.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

9 years agohtools: add JSON instance for EvacMode
Iustin Pop [Fri, 8 Jul 2011 12:19:17 +0000 (14:19 +0200)]
htools: add JSON instance for EvacMode

This abstracts the JSON parsing of the type EvacMode near its
definition, and simplifies its conversion in IAlloc.parseData.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

9 years agohtools: add human-readable output to hspace
Iustin Pop [Fri, 8 Jul 2011 11:53:14 +0000 (13:53 +0200)]
htools: add human-readable output to hspace

Currently, hspace can only output a machine-readable format that
(while detailed) is hard to parse quickly by people. This patch adds
(and enables by default) a human-readable output that shows the most
important metrics in a simple format.

Most of the work of the patch is in moving the display of various
metrics from the 'main' function to separate functions, each of which
can output either a machine or human intended format.

The patch also corrects a bug in the CPU efficiency display: before,
the efficiency was computed as instance virtual CPUs divided by total
physical CPUs, which is almost always supra-unitary. More correct is
to divide by the total virtual CPUs, which shows a more meaningful
number (when the p-to-v CPU ratio has been defined correctly).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

9 years agoFix job constants use in htools
Iustin Pop [Thu, 21 Jul 2011 11:35:36 +0000 (13:35 +0200)]
Fix job constants use in htools

Commit 56c094b4 added use of job constants, but I didn't pay
attention and ended up mixing things: job constants were used for
opcode ones, and the job ones didn't get converted.

This patch corrects it and uses only C.* constants throughout the Jobs
module.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

9 years agoAdd error state to LUGroupEvacuate's exceptions
Michael Hanselmann [Thu, 21 Jul 2011 09:53:29 +0000 (11:53 +0200)]
Add error state to LUGroupEvacuate's exceptions

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoRename *_STATUS_WAITLOCK to …_WAITING
Michael Hanselmann [Thu, 21 Jul 2011 09:23:28 +0000 (11:23 +0200)]
Rename *_STATUS_WAITLOCK to …_WAITING

This patch renames the {JOB,OP}_STATUS_WAITLOCK constants to
{JOB,OP}_STATUS_WAITING, as per design document for chained jobs.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agognt-group: Add command to evacuate whole group
Michael Hanselmann [Wed, 20 Jul 2011 11:39:58 +0000 (13:39 +0200)]
gnt-group: Add command to evacuate whole group

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoAdd new opcode for evacuating group
Michael Hanselmann [Tue, 17 May 2011 14:10:38 +0000 (16:10 +0200)]
Add new opcode for evacuating group

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoFix locking issue with job dependencies
Michael Hanselmann [Thu, 14 Jul 2011 22:55:20 +0000 (00:55 +0200)]
Fix locking issue with job dependencies

When jobs waiting for a dependency are notified, they're re-added to the
queue. This would require owning the queue lock in exclusive mode, but
since the function doing so is called from within the job/opcode
processor, it only holds the lock in shared mode.

This patch changes the result of the processor from a boolean to a
status value (integer). This way the caller can be notified about
actions to take, including notifying waiting jobs. The function adding
jobs to the queue can now acquire the lock in exclusive mode.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agojqueue: Read-only jobs don't need processor lock
Michael Hanselmann [Thu, 14 Jul 2011 21:31:33 +0000 (23:31 +0200)]
jqueue: Read-only jobs don't need processor lock

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoAdd support for KVM keymaps
Sébastien Bocahu [Wed, 20 Jul 2011 17:49:20 +0000 (19:49 +0200)]
Add support for KVM keymaps

Signed-off-by: Sébastien Bocahu <zecrazytux@zecrazytux.net>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

9 years agognt-debug: Add tests for job dependencies
Michael Hanselmann [Fri, 8 Jul 2011 01:43:23 +0000 (03:43 +0200)]
gnt-debug: Add tests for job dependencies

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agojqueue: Implement submitting multiple jobs with dependencies
Michael Hanselmann [Fri, 8 Jul 2011 21:49:03 +0000 (23:49 +0200)]
jqueue: Implement submitting multiple jobs with dependencies

With this change users of the “SubmitManyJobs” interface can use
relative job dependencies. Relative job IDs in dependencies are resolved
before handing the job off to the workerpool.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoFix node evacuation
Michael Hanselmann [Wed, 20 Jul 2011 11:18:41 +0000 (13:18 +0200)]
Fix node evacuation

- Adjust for new iallocator result format
- Split some code into helper functions

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoDo proper name lookup for the -O option
Guido Trotter [Fri, 15 Jul 2011 14:26:10 +0000 (14:26 +0000)]
Do proper name lookup for the -O option

hspace and hbal treat -O differently, and use aliases for short names
(although hbal succeeds in that, and hspace doesn't). Uniform this with
a name lookup, using the same functions we used for instance
selection/exclusion.

Some of the code is by the way a bit repetitive, and could probably be
merged in a single function. That needs to be a monadic one, though, so
I promise to do it as soon as I realize how to write them! ;)

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agojqueue: Add “writable” flag to memory objects
Michael Hanselmann [Thu, 14 Jul 2011 20:48:06 +0000 (22:48 +0200)]
jqueue: Add “writable” flag to memory objects

Basically only one instance of the job, the one being processed,
should be serialized to disk and replicated to other nodes. With
this flag assertions can be added in various places.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoImplement chained jobs
Michael Hanselmann [Wed, 1 Jun 2011 15:42:04 +0000 (17:42 +0200)]
Implement chained jobs

An overview is available in the design document for this change,
doc/design-chained-jobs.rst.

When a job enters the job processor, the current opcode's dependencies
are evaluated. If a referenced job has not yet reached the desired
status, the current job is registered as a dependant. The job processor
will continue to work on other pending tasks. When a job finishes it
notifies any pending dependants by re-adding them to the workerpool.

A per-job processor lock is necessary for rare cases where the same job
can be re-added twice.

There is no way to view waiting jobs at the moment, but I plan to
export this information to “gnt-debug locks”.

A so-called dependency manager takes care of managing waiting jobs and
keeping track of their status.

Unittests are included.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoAdd implementation details to design for chained jobs
Michael Hanselmann [Fri, 15 Jul 2011 21:45:04 +0000 (23:45 +0200)]
Add implementation details to design for chained jobs

As requested by Iustin.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoAdd support for GPT by using parted for disks bigger than 2TB.
Pedro Macedo [Tue, 19 Jul 2011 15:37:56 +0000 (17:37 +0200)]
Add support for GPT by using parted for disks bigger than 2TB.

Signed-off-by: Pedro Macedo <pmacedo@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

9 years agoRemove constants for iallocator multi-relocate
Michael Hanselmann [Fri, 15 Jul 2011 22:56:33 +0000 (00:56 +0200)]
Remove constants for iallocator multi-relocate

They're no longer necessary.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agohtools: add a machine-readable CLI flag
Iustin Pop [Fri, 8 Jul 2011 11:07:21 +0000 (13:07 +0200)]
htools: add a machine-readable CLI flag

This will be used in hspace to toggle between "human" readable
and machine readable output formats.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: move the '-p' option to htools.rst
Iustin Pop [Fri, 8 Jul 2011 13:29:42 +0000 (15:29 +0200)]
htools: move the '-p' option to htools.rst

Since this is a common option and has a big description.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

9 years agohtools: move tiered spec map helper to Hspace.hs
Iustin Pop [Fri, 8 Jul 2011 10:46:05 +0000 (12:46 +0200)]
htools: move tiered spec map helper to Hspace.hs

This is used just in hspace, so let's help in making Cluster.hs
smaller. We also split the function in two, as computing the spec map
and formatting it are two different tasks.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: import the program modules in QC.hs
Iustin Pop [Fri, 8 Jul 2011 09:57:06 +0000 (11:57 +0200)]
htools: import the program modules in QC.hs

This adds the binaries code to the coverage, and thus the coverage
finally shows the real coverage over all logic code (except for the
htools.hs code, which is not logic code related to the algorithms, so
it doesn't matter — plus it's also very small).

Next steps will be to actually add coverage for this code, especially
for hbal and hspace, which are relatively big compared to hail and
hscan (around 800 expressions versus 200-300 expressions).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: switch hspace to the generic binary
Iustin Pop [Fri, 8 Jul 2011 09:50:16 +0000 (11:50 +0200)]
htools: switch hspace to the generic binary

This is the last patch of the binaries conversion.

As information, we now have a single binary that is approx. 5.4MiB in
size, compared to 4 binaries that were approx. 5.1-5.2MiB in size;
this will result in a smaller package and install size, and the single
compilation phase should also help.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: switch hscan to the generic binary
Iustin Pop [Fri, 8 Jul 2011 09:45:10 +0000 (11:45 +0200)]
htools: switch hscan to the generic binary

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: switch hbal to the generic binary
Iustin Pop [Fri, 8 Jul 2011 09:41:01 +0000 (11:41 +0200)]
htools: switch hbal to the generic binary

In addition, the patch adds a separate Makefile variable for holding
the binary roles to make it more clear what we symlink.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: switch hail to the generic binary
Iustin Pop [Fri, 8 Jul 2011 09:33:52 +0000 (11:33 +0200)]
htools: switch hail to the generic binary

This converts the first binary to the generic 'htools' binary.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: add a generic binary
Iustin Pop [Thu, 7 Jul 2011 21:43:07 +0000 (23:43 +0200)]
htools: add a generic binary

This is the start of a series of patches that will unify all the
binaries currently in use in a single one, which can perform different
roles based on the name it is installed as.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: add a compatibility module
Iustin Pop [Thu, 7 Jul 2011 21:00:45 +0000 (23:00 +0200)]
htools: add a compatibility module

When compiling with the parallel-3.x library, we get a deprecation
warning, which makes understanding any other error messages
harder. This patch adds a compatibility module that will hold such
code for transitioning libraries.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: remove no-longer-needed tryMG* functionality
Iustin Pop [Thu, 7 Jul 2011 20:27:34 +0000 (22:27 +0200)]
htools: remove no-longer-needed tryMG* functionality

… which was deprecated by the previous patch.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: remove ialloc/relocate and multi-evacuate
Iustin Pop [Thu, 7 Jul 2011 20:23:23 +0000 (22:23 +0200)]
htools: remove ialloc/relocate and multi-evacuate

Since the new node-evacuate mode does both their work and also
supports better multi-group clusters (including handling split
instances).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: fix potential bug in ialloc/change-group
Iustin Pop [Thu, 7 Jul 2011 20:06:26 +0000 (22:06 +0200)]
htools: fix potential bug in ialloc/change-group

Currently, the ChangeAll mode of nodeEvac computes the primary group
of the instance and then uses the resulting group index for computing
the group score. However, during the change-group operation (which
also uses ChangeAll), the group of the instance at the beginning and
end of the operation differs, so we can't do that and we must instead
pass the final group to the function.

This patch does that, by passing the group from the caller of
nodeEvac.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: run IAllocator input through checkData
Iustin Pop [Thu, 7 Jul 2011 18:00:40 +0000 (20:00 +0200)]
htools: run IAllocator input through checkData

As the IAllocator backend is using a different data path than the
others, it doesn't get the full functionality that loadExternalData
does. This results in the current situation where checkData is not run
on the input cluster state, which means the node memory properties are
not correctly set, leading to no Xmem being set correctly.

This patch fixes this bug by adding a manual call to checkData.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: abstract a function for displaying warnings
Iustin Pop [Thu, 7 Jul 2011 17:59:45 +0000 (19:59 +0200)]
htools: abstract a function for displaying warnings

This will make it possible to reuse this in IAllocator too.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: use maybePrintNodes in hail.hs
Iustin Pop [Thu, 7 Jul 2011 17:13:04 +0000 (19:13 +0200)]
htools: use maybePrintNodes in hail.hs

This eliminates duplication of codes (and was forgotten back when
maybePrintNodes was added).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: add cluster state saving support to hail
Iustin Pop [Thu, 7 Jul 2011 17:09:12 +0000 (19:09 +0200)]
htools: add cluster state saving support to hail

This add support for saving the cluster state (both pre- and
post-iallocator run) to a text file such that it can be fed back into
any of the htools commands.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: return the final instance map in ialloc
Iustin Pop [Thu, 7 Jul 2011 16:59:04 +0000 (18:59 +0200)]
htools: return the final instance map in ialloc

Similar to the previous patch, this returns the final instance map
from the iallocator run, which will allow saving the cluster state for
further examination/post-processing.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: implement post-alloc cluster status display
Iustin Pop [Thu, 7 Jul 2011 16:48:30 +0000 (18:48 +0200)]
htools: implement post-alloc cluster status display

This patch changes the IAllocator result formatting workflow to return
the final node list, which can be then used to display the final node
status too—currently only the initial status can be shown, which is
only half useful.

Note that as the FIXME in the code says, doing this right for the
evacuate mode is hard; however, as that mode is deprecated, we can
live it for the moment.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agoUpdate node group iallocator design to use job dependencies
Michael Hanselmann [Mon, 11 Jul 2011 19:02:58 +0000 (21:02 +0200)]
Update node group iallocator design to use job dependencies

While working on a function to submit jobsets, I realized that we
actually don't need them anymore. With the new job dependencies, the
iallocator plugin can just generate the right dependencies and gets the
same effect as with jobsets. This saves us from having to implement
jobsets in Ganeti core.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoFix assertion error on unclean master shutdown
Michael Hanselmann [Thu, 14 Jul 2011 20:49:34 +0000 (22:49 +0200)]
Fix assertion error on unclean master shutdown

Commit 66bd7445 added an assertion to ensure a finalized job has its
“end_timestamp” attribute set. Unfortunately it didn't cover a case when
the queue is recovering from an unclean master shutdown.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoMake SharedLock._is_owned public
Michael Hanselmann [Wed, 13 Jul 2011 23:51:45 +0000 (01:51 +0200)]
Make SharedLock._is_owned public

This will be useful for assertions. GanetiLockManager._is_owned is
exported, too.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agohtools: return new state from new IAllocator modes
Iustin Pop [Thu, 7 Jul 2011 16:36:24 +0000 (18:36 +0200)]
htools: return new state from new IAllocator modes

The old modes already return the node list (as part of AllocSolution),
this patch makes the new modes provide this new information.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: update the node-evac/change-group result
Iustin Pop [Thu, 7 Jul 2011 13:45:57 +0000 (15:45 +0200)]
htools: update the node-evac/change-group result

… per the new design changes. This also requires changes to
updateEvacSolution, as passing in the original instance is not
helpful; we just pass the instance index now and look up the actual
instance in the modified instance map.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: add function for getting all nodes of an instance
Iustin Pop [Thu, 7 Jul 2011 13:45:05 +0000 (15:45 +0200)]
htools: add function for getting all nodes of an instance

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: add change-group IAllocator mode
Iustin Pop [Thu, 7 Jul 2011 12:11:36 +0000 (14:11 +0200)]
htools: add change-group IAllocator mode

This builds a lot upon the node-evacuation mode, and its
implementation is therefore quite easy (once node-evac was added).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: add group restrictions to findBestAllocGroup
Iustin Pop [Thu, 7 Jul 2011 11:58:30 +0000 (13:58 +0200)]
htools: add group restrictions to findBestAllocGroup

While a normal multi-group allocation doesn't need to restrict the
list of target groups beyond the unallocable status of some groups,
when we relocate instances from one group to the other we need to
limit the subset of searched groups.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: generalise and rename availableLocalNodes
Iustin Pop [Thu, 7 Jul 2011 11:35:48 +0000 (13:35 +0200)]
htools: generalise and rename availableLocalNodes

Currently this function works in the context of an instance, computing
its local group nodes. We can use generalise it to take a group index
argument, so that it's not bound to an instance, and we rename it
accordingly.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: add ChangeGroup to IAllocator types/loader
Iustin Pop [Wed, 6 Jul 2011 22:57:14 +0000 (00:57 +0200)]
htools: add ChangeGroup to IAllocator types/loader

This patch updates the types and functions in IAllocator.hs and
Loader.hs to the new design changes (elimination of multi-relocate
mode and replacement with change-group).

It also removes an extra re-export of EvacMode from Loader.hs.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: split tryMGAlloc in two
Iustin Pop [Wed, 6 Jul 2011 22:44:21 +0000 (00:44 +0200)]
htools: split tryMGAlloc in two

Currently, this function does both identify the best group for an
instance, and format the results (partially). The former functionality
is useful in other contexts too, so let's split the function in two.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: add node-evacuation of DRBD all nodes
Iustin Pop [Wed, 6 Jul 2011 22:13:27 +0000 (00:13 +0200)]
htools: add node-evacuation of DRBD all nodes

This is much more complicated than either change primary (which is
trivial) and change secondary (which is also simple).

The main strategy is:

- choose a new secondary (that will become the final primary) and
  replace-disks to it
- failover
- choose a new secondary again (this will be the final secondary)

However, if the original primary was offline, then we first have to
failover to the secondary of the instance.

Note: currently this evacuation mode, while producing correct
evacuation results, does not result in optimal placement of the
instance; it will need attention and possibly an algorithm change.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: small optimisation in Cluster.collapseFailures
Iustin Pop [Wed, 6 Jul 2011 21:29:57 +0000 (23:29 +0200)]
htools: small optimisation in Cluster.collapseFailures

This makes fewer temporary data structures, and thus brings a bit of
improvement in total runtime.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: add optional limit to allocation count
Iustin Pop [Wed, 6 Jul 2011 20:30:42 +0000 (22:30 +0200)]
htools: add optional limit to allocation count

Currently, the tieredAlloc/iterateAlloc functions will not return
until the allocation fails; this means unit-testing their
functionality (e.g. that an instance can be allocated) is slow, since
they will allocate all possible instances.

This patch adds an optional limit that allows allocation to return
early; this makes the cluster unittests twice as fast.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: add node-evacuation of DRBD secondaries
Iustin Pop [Wed, 6 Jul 2011 14:29:14 +0000 (16:29 +0200)]
htools: add node-evacuation of DRBD secondaries

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: rework compCV so that we can use [Node] too
Iustin Pop [Wed, 6 Jul 2011 14:52:01 +0000 (16:52 +0200)]
htools: rework compCV so that we can use [Node] too

In some cases, we have a [Node.Node] list, instead of a Node.List
structure. Instead of converting [Node.Node] to the latter, let's
rework compCV and add a compCVNodes such that we skip this
(potentially expensive) step.

This already allows a simplification in a unittest, and will simplify
consumers of Node.computeGroups.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agoAdding a wrapper around connecting to kvm console
Stephen Shirley [Fri, 24 Jun 2011 11:29:48 +0000 (13:29 +0200)]
Adding a wrapper around connecting to kvm console

The wrapper will connect to the console, and check in the background if
the instance is paused, unpausing it as necessary.

Signed-off-by: Stephen Shirley <diamond@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

9 years agoAdding a wrapper around "xm console"
Stephen Shirley [Thu, 23 Jun 2011 09:42:23 +0000 (11:42 +0200)]
Adding a wrapper around "xm console"

The wrapper will connect to the console, and check in the background if
the instance is paused, unpausing it as necessary.

Signed-off-by: Stephen Shirley <diamond@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

9 years agoAdd opcode attribute for chained jobs
Michael Hanselmann [Fri, 27 May 2011 10:50:15 +0000 (12:50 +0200)]
Add opcode attribute for chained jobs

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoDesign for chained jobs
Michael Hanselmann [Wed, 18 May 2011 14:38:12 +0000 (16:38 +0200)]
Design for chained jobs

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agohtools: fix a doc bug in the hbal man page
Iustin Pop [Fri, 8 Jul 2011 13:32:23 +0000 (15:32 +0200)]
htools: fix a doc bug in the hbal man page

Similar to the previous hspace fix.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

9 years agoMark htools/README as obsolete
Iustin Pop [Fri, 8 Jul 2011 14:01:49 +0000 (16:01 +0200)]
Mark htools/README as obsolete

And point to the main INSTALL file for up-to-date information.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

9 years agoFix htool unittest failure due to invalid prereqs
Iustin Pop [Thu, 14 Jul 2011 09:32:22 +0000 (11:32 +0200)]
Fix htool unittest failure due to invalid prereqs

Currently, the test prop_Node_rMem does some reserved memory checks on
nodes, and for the test to work correctly (can add more instances) we
require that the node free memory is greater than zero via a
prerequisite.

However, if the node free memory is exactly 1, the call to
setInstanceSmallerThanNode will set the instance memory to "free
memory `div` 2", and since this is integer division, the instance
memory will get set to 0. Which means that the node reserved memory
will not increase when we add this instance as secondary.

To work around, we simply require that the node memory is high enough
(we use our 'base' unit of memory).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: add node-evacuate of DRBD8 primaries
Iustin Pop [Tue, 5 Jul 2011 21:32:19 +0000 (23:32 +0200)]
htools: add node-evacuate of DRBD8 primaries

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: add more IAlloc data types and functions
Iustin Pop [Sun, 3 Jul 2011 14:17:46 +0000 (16:17 +0200)]
htools: add more IAlloc data types and functions

These move the 'not supported' message deeper into the workflow;
assuming that the workflow as added by this patch is correct, all that
remains is:

- for disk template DRBD8, plug in the currently existing mechanisms
  to implement relocation
- for other disk templates, first modify the Node function to handle
  node changes correctly for each disk template, and then add support
  for them in nodeEvacInstance

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: add two monad-conversion utility functions
Iustin Pop [Wed, 6 Jul 2011 22:12:28 +0000 (00:12 +0200)]
htools: add two monad-conversion utility functions

For monadic operations, these function will make it easier to convert
other monads (or monad-like structure) to our 'Result' type.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: add a function for computing evacuated nodes
Iustin Pop [Mon, 4 Jul 2011 21:41:52 +0000 (23:41 +0200)]
htools: add a function for computing evacuated nodes

The new IAllocator interface is based on instances, not nodes, so we
need to backtrack and compute on which nodes we can't allocate
instances during the current operation. This patch adds a function for
computing this node set.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: rename an internal constant
Iustin Pop [Mon, 4 Jul 2011 12:31:04 +0000 (14:31 +0200)]
htools: rename an internal constant

'emptySolution' is fine while we have only one Solution type
(AllocSolution), but in the future we'll introduce another one and
thus we should rename it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: abstract and simplify a multigroup function
Iustin Pop [Sun, 3 Jul 2011 21:16:06 +0000 (23:16 +0200)]
htools: abstract and simplify a multigroup function

In the future, this 'split/associated instance idx list' functionality
will be used by other code, so let's move it to its own code.

The new method of computing the split also requires exporting one more
function from IntMap in Container.hs.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohspace: also display the starting tiered specs
Iustin Pop [Sun, 3 Jul 2011 13:53:57 +0000 (15:53 +0200)]
hspace: also display the starting tiered specs

These are useful for checking what instance specifications we start
with (as they are not necessarily the first entry in the TSPEC map).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohspace: add support for units in the options
Iustin Pop [Sun, 3 Jul 2011 13:46:10 +0000 (15:46 +0200)]
hspace: add support for units in the options

This patch converts hspace to allow units in all disk and memory
specifications, and updates the man page accordingly. It also correct
a mistake in the man page for the --max-cpu option.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: add parseUnit function
Iustin Pop [Sun, 3 Jul 2011 13:18:42 +0000 (15:18 +0200)]
htools: add parseUnit function

This is similar, but not identical, to Utils.ParseUnit. The biggest
difference is that we don't round up/down; as we only use integral
types, the result will always be rounded down.

Moreover, since (real-world) disk sizes come in SI units, the function
differentiates between SI and binary prefixes, using lower-case for
binary and upper-case for SI (similar to lvm usage). This distinction
should be ported to the Python code.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

9 years agohtools: simplify select tests
Guido Trotter [Wed, 13 Jul 2011 15:03:01 +0000 (15:03 +0000)]
htools: simplify select tests

Rather than running all possible tests in one go, split them out in
three cases: "normal", undefined default value, undefined list values
(both of which require a valid response before them).

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

9 years agoSet startup_paused to False when restarting
Stephen Shirley [Wed, 13 Jul 2011 15:58:50 +0000 (17:58 +0200)]
Set startup_paused to False when restarting

This fixes the lint error:

E1120:1220:InstanceReboot: No value passed for parameter
'startup_paused' in function call

Signed-off-by: Stephen Shirley <diamond@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>