ganeti-github.git
8 years agoSwitch devel/upload to a static file
Iustin Pop [Wed, 7 Nov 2012 11:49:11 +0000 (12:49 +0100)]
Switch devel/upload to a static file

We had twice in the past days questions about devel/upload being
"broken", since bash re-reads shell scripts during their run and this
file can get regenerated due to Makefile changes.

Since we only need this to be dynamically built for 3 variables, let's
make the file static and read those three variables when it is run,
instead of when it is built, which allows us to re-read the "latest
version" of these vars as well.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agoUse SSH_LOGIN_USER rather than root for xl ssh
Guido Trotter [Tue, 6 Nov 2012 17:52:59 +0000 (18:52 +0100)]
Use SSH_LOGIN_USER rather than root for xl ssh

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoAdd utility to check if file is executable
Michael Hanselmann [Tue, 6 Nov 2012 17:27:39 +0000 (18:27 +0100)]
Add utility to check if file is executable

This replaces direct calls to “os.access” and
“os.path.exists”/“os.path.isfile”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoFix NameError in constants.py introduced in merge 46c1f82
Michael Hanselmann [Tue, 6 Nov 2012 17:12:11 +0000 (18:12 +0100)]
Fix NameError in constants.py introduced in merge 46c1f82

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoAdd test for Luxi calls consistency hs/py
Iustin Pop [Tue, 6 Nov 2012 15:47:53 +0000 (16:47 +0100)]
Add test for Luxi calls consistency hs/py

This tests that the same Luxi calls are defined in Python and
Haskell. It doesn't test yet that their serialisation is correct
though.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

8 years agoMerge branch 'devel-2.6'
Guido Trotter [Tue, 6 Nov 2012 16:22:40 +0000 (17:22 +0100)]
Merge branch 'devel-2.6'

* devel-2.6:
  Disable E1101 on ganeti/http/server.py:424
  Fix live migration under xl
  Don't check for xend port when using xl

Conflicts:
lib/hypervisor/hv_xen.py: trivial

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoMerge branch 'stable-2.6' into devel-2.6
Guido Trotter [Tue, 6 Nov 2012 16:09:40 +0000 (17:09 +0100)]
Merge branch 'stable-2.6' into devel-2.6

* stable-2.6:
  Disable E1101 on ganeti/http/server.py:424
  Fix live migration under xl
  Don't check for xend port when using xl

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoDisable E1101 on ganeti/http/server.py:424
Guido Trotter [Tue, 6 Nov 2012 15:35:34 +0000 (16:35 +0100)]
Disable E1101 on ganeti/http/server.py:424

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoFix live migration under xl
Guido Trotter [Mon, 5 Nov 2012 14:18:33 +0000 (15:18 +0100)]
Fix live migration under xl

Until now the only way to make live migration work in conjunction with
"xl" was to add ssh known_hosts keys for every node's secondary ip on
every other node.

With this command we remove the target key verification: this is not
worse than what we were doing before with "xm", and allows the migration
to happen under either toolstack, without extra manual work. Of course
the full security of ssh is not used by live migration, then.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agoDon't check for xend port when using xl
Guido Trotter [Mon, 5 Nov 2012 13:32:24 +0000 (14:32 +0100)]
Don't check for xend port when using xl

If the toolstack is set to "xl" we shouldn't ping xend for liveness
before attempting a live migration.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoutils.io: Improve handling of double and single slashes
Michael Hanselmann [Mon, 5 Nov 2012 18:12:51 +0000 (19:12 +0100)]
utils.io: Improve handling of double and single slashes

Up until now “IsBelowDir("/", …)” would never return True. The reason
was that an additional slash was added to the root path resulting in
“//", which is “implementation-defined” in posix and treated specially
by “os.path.normpath”.

This patch fixes the behaviour for this special case and adds tests
(also for IsNormAbsPath). A typo in the docstring is fixed. Calls to
“assert_” and “assertFalse” are changed to pass a message by keyword
argument.

It is a bit of a mess, but I hope the resulting behaviour is correct.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Dato Simó <dato@google.com>

8 years agoworkerpool: Don't mask variable in AddManyTasks
Michael Hanselmann [Fri, 2 Nov 2012 17:07:28 +0000 (18:07 +0100)]
workerpool: Don't mask variable in AddManyTasks

The name “priority” is already used.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

8 years agoworkerpool: Simplify _WaitForTaskUnlocked
Michael Hanselmann [Tue, 30 Oct 2012 15:43:24 +0000 (16:43 +0100)]
workerpool: Simplify _WaitForTaskUnlocked

The function in is simplified in its structure and duplicated checks
have been merged.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

8 years agocli.py: use None as name for tag operations on the cluster
Dato Simó [Mon, 5 Nov 2012 17:54:59 +0000 (17:54 +0000)]
cli.py: use None as name for tag operations on the cluster

This change is mostly cosmetic. Previously, the literal "cluster" was
used for the 'name' field of tag operations on the cluster (as opposed
to a node or an instance). Since this field has a type of TMaybeString
specifically for the case of the cluster, it seems more correct to use
None, rather than an arbitrary string (that is not used by the callee).

Additionally: note in opcodes.py that groups also expect a name; the
previous comment only referred to nodes and instances.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agoFix previous merge
Bernardo Dal Seno [Thu, 1 Nov 2012 20:45:18 +0000 (21:45 +0100)]
Fix previous merge

A call to _CalculateGroupIPolicy wasn't refactored during the merge.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agoMerge branch 'devel-2.6'
Michael Hanselmann [Thu, 1 Nov 2012 17:27:27 +0000 (18:27 +0100)]
Merge branch 'devel-2.6'

* devel-2.6:
  Fix typo in gnt-instance man page
  jqueue: Return jobs to queue when shutting down
  gnt-debug delay: Add "--submit" option
  Make hostname checks uniform between instance rename and add
  Improve logging of new job submissions
  Improve handling of lock exceptions
  Add note about developing on a production machine
  Fix runtime memory increases
  Fix validation of vgname in OpClusterSetParams
  Fix removal of storage directory on shared file storage
  Switch non-redundant check to disk template-based
  Document the new --yes-do-it option for master-failover
  Add option to force master-failover without voting
  Update instance modify message
  Force tap's MAC prefix to "fe"
  Fix disk adoption interaction with ipolicy checks
  Better NEWS file for 2.6.1

Conflicts:
lib/cmdlib.py: Trivial
tools/kvm-ifup.in: Space vs. tab

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoMerge branch 'stable-2.6' into devel-2.6
Michael Hanselmann [Thu, 1 Nov 2012 15:59:01 +0000 (16:59 +0100)]
Merge branch 'stable-2.6' into devel-2.6

* stable-2.6:
  Fix typo in gnt-instance man page
  jqueue: Return jobs to queue when shutting down
  gnt-debug delay: Add "--submit" option
  Make hostname checks uniform between instance rename and add
  Improve logging of new job submissions
  Improve handling of lock exceptions
  Add note about developing on a production machine
  Fix runtime memory increases
  Fix validation of vgname in OpClusterSetParams
  Fix removal of storage directory on shared file storage
  Switch non-redundant check to disk template-based
  Document the new --yes-do-it option for master-failover
  Add option to force master-failover without voting
  Update instance modify message
  Force tap's MAC prefix to "fe"
  Fix disk adoption interaction with ipolicy checks
  Better NEWS file for 2.6.1

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoFix typo in gnt-instance man page
Michael Hanselmann [Thu, 1 Nov 2012 15:04:33 +0000 (16:04 +0100)]
Fix typo in gnt-instance man page

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agojqueue: Return jobs to queue when shutting down
Michael Hanselmann [Thu, 1 Nov 2012 13:54:03 +0000 (14:54 +0100)]
jqueue: Return jobs to queue when shutting down

When a job is still waiting for locks and the queue is shutting down,
they should be returned and not actually start processing. Until now
jobs which transitioned from “queued” to “waiting” were already
considered to be running as far as the shutdown code was concerned.

This fixes issue 296.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agognt-debug delay: Add "--submit" option
Michael Hanselmann [Thu, 1 Nov 2012 14:23:54 +0000 (15:23 +0100)]
gnt-debug delay: Add "--submit" option

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoRemove duplicate workerpool test
Michael Hanselmann [Wed, 31 Oct 2012 13:17:09 +0000 (14:17 +0100)]
Remove duplicate workerpool test

Commit 52c47e4e (July 2010) added the exact test twice, probably due to
a copy & paste error.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoMake hostname checks uniform between instance rename and add
Iustin Pop [Wed, 31 Oct 2012 16:10:19 +0000 (17:10 +0100)]
Make hostname checks uniform between instance rename and add

Currently, we have instance rename doing extra checks on the host
name, to prevent accidental wrong renames; however, instance create
doesn't do these checks (issue 291), which (if DNS is misconfigured)
can lead to hard to diagnose errors.

This patch abstracts the name checking from LUInstanceRename into a
separate function, which is then reused in both instance rename and
instance create.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

8 years agoImprove logging of new job submissions
Iustin Pop [Wed, 31 Oct 2012 13:47:18 +0000 (14:47 +0100)]
Improve logging of new job submissions

This addresses issue 290: when receiving new jobs, logging is
incomplete, and we don't have the job ID(s) and/or summaries
logged. Only later, when the job is queried for or being processed, we
know more.

This is not good when troubleshooting, so let's improve the initial
logging.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

8 years agoImprove handling of lock exceptions
Iustin Pop [Wed, 31 Oct 2012 13:31:48 +0000 (14:31 +0100)]
Improve handling of lock exceptions

There are two issues with lock exceptions right now:

- first, we don't log the original error; this is fine for now
  (locking.py always returns the same error here), but in general is
  brittle: if locking.py would start returning more information, we'd
  completely miss that

- second, an actual honest lock conflict is not an internal error;
  it's simply an optimistic lock failing, and as such we should not
  return internal error, but rather resource_not_unique

This addresses issue 287.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

8 years agoAdd note about developing on a production machine
Iustin Pop [Tue, 30 Oct 2012 15:49:33 +0000 (16:49 +0100)]
Add note about developing on a production machine

This is the bit of documentation missing for issue 170. Doing
development on a machine which already has Ganeti installed kind of
works, but only when the installed and the developed version are very
similar, and even then it can be problematic.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agoFix runtime memory increases
Iustin Pop [Tue, 30 Oct 2012 13:43:09 +0000 (14:43 +0100)]
Fix runtime memory increases

Commit 2c0af7da which added the runtime memory changes functionality
had a small typo (wrong name); I've rewritten this to only compute the
delta once, for simplicity.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

8 years agoFix validation of vgname in OpClusterSetParams
Iustin Pop [Tue, 30 Oct 2012 11:53:26 +0000 (12:53 +0100)]
Fix validation of vgname in OpClusterSetParams

This variable can be empty, when we want to disable LVM, so we can't
use TMaybeString.

Fixes issue 285.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

8 years agoFix removal of storage directory on shared file storage
Iustin Pop [Tue, 30 Oct 2012 11:29:16 +0000 (12:29 +0100)]
Fix removal of storage directory on shared file storage

This patch makes _RemoveDisks symmetric to _CreateDisks with respect
to file-based storage: _CreateDisks uses "in constants.DTS_FILEBASED",
whereas _RemoveDisks was not update and only uses "==
constants.DT_FILE". This results in stale directories left on the
filesystem.

Fixes issue 262.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

8 years agoSwitch non-redundant check to disk template-based
Iustin Pop [Tue, 30 Oct 2012 11:14:37 +0000 (12:14 +0100)]
Switch non-redundant check to disk template-based

Currently, the warning/notice about non-redundant instances in cluster
verify is based non empty secondaries list (how old is this?); the
proper way to check this nowadays is via DTS_MIRRORED.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

8 years agoFix permission for socket directory
Bernardo Dal Seno [Mon, 29 Oct 2012 17:44:35 +0000 (18:44 +0100)]
Fix permission for socket directory

The directory must we writable also by the confd daemon user.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoDocument the new --yes-do-it option for master-failover
Iustin Pop [Mon, 29 Oct 2012 12:01:58 +0000 (13:01 +0100)]
Document the new --yes-do-it option for master-failover

Sorry, this should have gone in the previous commit, I forgot about
it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

8 years agoAdd option to force master-failover without voting
Iustin Pop [Mon, 29 Oct 2012 11:21:59 +0000 (12:21 +0100)]
Add option to force master-failover without voting

This fixes issue 282.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

8 years agoConvert Luxi results to Ganeti errors
Iustin Pop [Fri, 26 Oct 2012 13:27:11 +0000 (15:27 +0200)]
Convert Luxi results to Ganeti errors

This a bit too complex patch converts the result of Luxi calls
(submitJob, query*, etc.) from Result to ErrorResult. It then
immediately revers this in the HTools/Backend/Luxi module, where we
don't need necessarily the full error type (just a nice error
message), and does the same in Hbal's job execution functions.

While at first sight this doesn't seem to do much, what we get is
actual error messages from Ganeti, plus improvements to the result
parsing: instead of "can't parse char", we now get properly (note,
wrapped manually):

  Executing jobset for instances instance1, …
  Job submission error: Failure: the job queue is marked for drain and
    doesn't accept new requests

Or:

  Job submission error: Unhandled exception: LuxiError "parsing job
    id: cannot parse string 'a956101'"

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agocfgupgrade: Write file for file storage paths
Michael Hanselmann [Fri, 5 Oct 2012 01:32:44 +0000 (03:32 +0200)]
cfgupgrade: Write file for file storage paths

When file storage is used this file is now mandatory.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agobackend: Switch to new file storage directory verification
Michael Hanselmann [Fri, 5 Oct 2012 00:59:12 +0000 (02:59 +0200)]
backend: Switch to new file storage directory verification

The configuration is no longer used for verifying file storage paths.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoCheck allowed file storage paths during cluster-verify
Michael Hanselmann [Fri, 5 Oct 2012 00:43:47 +0000 (02:43 +0200)]
Check allowed file storage paths during cluster-verify

Some paths, such as /bin or /usr/lib, should not be used for file
storage. This patch implements a check during cluster verification to
fail in case such a path has been used.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoMake Paramiko an optional dependency for listrunner
Michael Hanselmann [Wed, 24 Oct 2012 00:01:42 +0000 (02:01 +0200)]
Make Paramiko an optional dependency for listrunner

With the move away from “setup-ssh”, Paramiko is no longer necessary to
configure SSH on nodes.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoRemove setup-ssh
Michael Hanselmann [Tue, 23 Oct 2012 23:58:37 +0000 (01:58 +0200)]
Remove setup-ssh

It has been superseeded by “prepare-node-join”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agognt-node add: Use prepare-node-join
Michael Hanselmann [Tue, 23 Oct 2012 18:16:25 +0000 (20:16 +0200)]
gnt-node add: Use prepare-node-join

This patch changes “gnt-node add” to use the newly added
“prepare-node-join” tool. Hereby Paramiko is no longer a hard dependency
for setting up SSH on nodes.

In “gnt_cluster.py”, a positional parameter is no longer passed as a
keyword parameter.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoprepare-node-join: Use ssh.GetAllUserFiles
Michael Hanselmann [Tue, 23 Oct 2012 23:16:50 +0000 (01:16 +0200)]
prepare-node-join: Use ssh.GetAllUserFiles

Instead of building the dictionary locally, the global version in
“ssh.py” can be used.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agossh: Add function to get all of user's SSH files
Michael Hanselmann [Tue, 23 Oct 2012 23:10:36 +0000 (01:10 +0200)]
ssh: Add function to get all of user's SSH files

This new function returns the file paths for all of a user's SSH-related
files (RSA, DSA and authorized_keys).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoRunCmd: Support standard input file descriptor
Michael Hanselmann [Tue, 23 Oct 2012 19:25:30 +0000 (21:25 +0200)]
RunCmd: Support standard input file descriptor

This patch changes “utils.RunCmd” to accept a file-like object or a
numeric file descriptor which will be used as the command's standard
input. One use-case will be to pass all necessary data to
“prepare-node-join”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoFactorize job selection in “gnt-job cancel”
Michael Hanselmann [Thu, 25 Oct 2012 15:29:51 +0000 (17:29 +0200)]
Factorize job selection in “gnt-job cancel”

This will also be used for changing jobs' priorities. All parameters to
the common function are non-optional.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoutils.x509: Factorize code to extract X509 certificate
Michael Hanselmann [Tue, 23 Oct 2012 23:55:53 +0000 (01:55 +0200)]
utils.x509: Factorize code to extract X509 certificate

This will be useful in “gnt-node add”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoprepare_node_join: Move daemon SSH files to constants
Michael Hanselmann [Tue, 23 Oct 2012 23:11:45 +0000 (01:11 +0200)]
prepare_node_join: Move daemon SSH files to constants

This dictionary will also be useful in “gnt-node add”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoprepare-node-join: Swap private and public keys
Michael Hanselmann [Tue, 23 Oct 2012 23:24:19 +0000 (01:24 +0200)]
prepare-node-join: Swap private and public keys

Other places, such as “ssh.GetUserFiles”, use a structure where the
private key comes before the private key. Until now prepare-node-join
did the opposite, that is the public key came first. To avoid confusion
and potential bugs, this is changed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoprepare-node-join: Use public key directly for auth…_keys
Michael Hanselmann [Tue, 23 Oct 2012 22:26:16 +0000 (00:26 +0200)]
prepare-node-join: Use public key directly for auth…_keys

A public key already includes the necessary prefix (“ssh-rsa” or
“ssh-dss”), so there is no need to add it again.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agossh.GetUserFiles: Parameter to disable directory check
Michael Hanselmann [Tue, 23 Oct 2012 22:55:39 +0000 (00:55 +0200)]
ssh.GetUserFiles: Parameter to disable directory check

Without this parameter, either an error would be raised or “.ssh” would
have to be created. Now it is possible to retrieve the paths without
requiring the “.ssh” directory to exist.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoMove htools backends to a separate directory
Iustin Pop [Fri, 26 Oct 2012 07:32:35 +0000 (09:32 +0200)]
Move htools backends to a separate directory

Five modules under the HTools/ directories are backend
implementations, so let's move them to a separate directory, to more
clearly show the hierarchy. I wanted to do this for a while, but
merging between branches is always an issue, so let's do it know since
we have an opportunity.

This patch contains the actual renames, the required changed module
names, imports, etc., but no other changes.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agoFix lint issue in Test/…/THH.hs
Iustin Pop [Fri, 26 Oct 2012 11:48:45 +0000 (13:48 +0200)]
Fix lint issue in Test/…/THH.hs

Sorry!

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agoFix X509CertError definition in Haskell codebase
Iustin Pop [Fri, 26 Oct 2012 08:22:21 +0000 (10:22 +0200)]
Fix X509CertError definition in Haskell codebase

Thanks Dato for catching this.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agoFix a few issues found by newer hlint
Iustin Pop [Fri, 26 Oct 2012 06:49:48 +0000 (08:49 +0200)]
Fix a few issues found by newer hlint

Testing with a newer hlint found a few minor issues; but all are real,
valid recommendations:

- don't use "if cond then f x else f y", but "f (if cond then x else y)"
- "if a then b else True" is equivalent to the simpler "not a || b"
- and as usual, one more ignore to our "testing basic properties"
  module

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agoAdd a simple unittest for THH code
Iustin Pop [Thu, 25 Oct 2012 14:04:56 +0000 (16:04 +0200)]
Add a simple unittest for THH code

This is very THH specific, and applies to all serialisations generated
by THH, so I'm adding it in its own module.

Probably we should add some more generic tests, but in general THH
code is tested by the various definitions; this new field type however
is not (yet), so this is why I want this specific unittest.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

8 years agoAdd support for optional fields with null serialised
Iustin Pop [Thu, 25 Oct 2012 13:48:05 +0000 (15:48 +0200)]
Add support for optional fields with null serialised

This follows a conversation we had for how to deal with
optional-but-required fields in JSON serialisations: fields which are
optional (can be either a given type or 'null'), but where the 'null'
value is required. There are just a few of these in the Python code,
but we should support them nevertheless.

The patch changes the 'isOptional' attribute from boolean to a custom
ADT, three-typed. This allows us to keep the same path on load (which
deals with both cases), but use a custom save path where we explicitly
save the 'null' value.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

8 years agoUpdate instance modify message
Iustin Pop [Fri, 26 Oct 2012 06:41:07 +0000 (08:41 +0200)]
Update instance modify message

Currently the message does not say explicitly that instance-initiated
reboots are useless to trigger the use of new parameters, per the
thread on the user mailing list. Let's improve it a bit.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

8 years agoErrors.hs: improve field names for ConfigVersionMismatch
Dato Simó [Thu, 25 Oct 2012 16:11:12 +0000 (17:11 +0100)]
Errors.hs: improve field names for ConfigVersionMismatch

Change {exp,act}Code to {exp,act}Ver, which gives a better idea that
the integer fields represent version numbers.

Also:

  - errors.py: update OpPrereqError's docstring to note that an error
    code is always expected as the second argument (it was previously
    optional).

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoRemove unused cache implementation
Iustin Pop [Thu, 25 Oct 2012 14:21:25 +0000 (16:21 +0200)]
Remove unused cache implementation

Note that this commit has no Makefile.am changes, as the files were
not actually used. So it's better to actually remove them.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

8 years agoTHH.hs: delete isOptional, no longer used
Dato Simó [Wed, 24 Oct 2012 14:28:57 +0000 (15:28 +0100)]
THH.hs: delete isOptional, no longer used

The isOptional function is no longer used after a150585 (“Convert
opcode TH code to the use of Field type”).

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoFix two typos in Ganeti administrator's guide
Michele Tartara [Tue, 23 Oct 2012 15:55:14 +0000 (17:55 +0200)]
Fix two typos in Ganeti administrator's guide

Fix a verb tense and add a missing verb.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

8 years agobdev: Remove unused import of itertools
Michael Hanselmann [Thu, 25 Oct 2012 12:54:53 +0000 (14:54 +0200)]
bdev: Remove unused import of itertools

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>

8 years agobdev: Add verification for file storage paths
Michael Hanselmann [Wed, 17 Oct 2012 14:10:07 +0000 (16:10 +0200)]
bdev: Add verification for file storage paths

An earlier version of this patch series verified all paths in cmdlib in
the master daemon. With this change all that verification code is moved
to bdev to run inside the node daemon. The checks are much stricter
now--it is no longer possible to use forbidden paths (e.g. /bin) to
manipulate file storage devices (once these checks are being used).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agojqueue: Factorize code to modify job
Michael Hanselmann [Wed, 24 Oct 2012 01:19:20 +0000 (03:19 +0200)]
jqueue: Factorize code to modify job

A new function will be added to change a job's priority.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

8 years agocli: Use callback for --priority
Michael Hanselmann [Wed, 24 Oct 2012 02:08:42 +0000 (04:08 +0200)]
cli: Use callback for --priority

If the option is used elsewhere, the numeric value is directly
available.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

8 years agojqueue: Add docstring for _DetermineJobDirectories
Michael Hanselmann [Wed, 24 Oct 2012 00:37:34 +0000 (02:37 +0200)]
jqueue: Add docstring for _DetermineJobDirectories

Somehow this was missed in commit 0422250e.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

8 years agojqueue: Fix comments in _SubmitJobUnlocked
Michael Hanselmann [Wed, 24 Oct 2012 00:38:27 +0000 (02:38 +0200)]
jqueue: Fix comments in _SubmitJobUnlocked

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

8 years agoDrop SSHS_FORCE constant
Michael Hanselmann [Tue, 23 Oct 2012 21:52:44 +0000 (23:52 +0200)]
Drop SSHS_FORCE constant

It is not actually used.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

8 years agoAdd a default `.ghci' file
Iustin Pop [Thu, 25 Oct 2012 10:33:10 +0000 (12:33 +0200)]
Add a default `.ghci' file

This options file for ghci preseeds the correct include paths, so that
interactive sessions don't need to always pass these args.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agoForce tap's MAC prefix to "fe"
Simon Deziel [Fri, 11 May 2012 16:00:24 +0000 (12:00 -0400)]
Force tap's MAC prefix to "fe"

Setting a high prefix discourages the bridge from adopting the tap's
MAC. Xen is not affected by this since the MAC is forced to
"fe:ff:ff:ff:ff:ff".

This addresses issue #217.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
(cherry picked from commit be3f52f304767a719d4b0dc07c81a1aaea8432b0)
(this is critical enough that we want it on stable-2.6)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

8 years agoConvert query path from string errors to GanetiException
Iustin Pop [Fri, 12 Oct 2012 12:37:20 +0000 (14:37 +0200)]
Convert query path from string errors to GanetiException

This patch converts all the call paths from 'Result' (which contains
just string errors) to 'ErrorResult', which holds
GanetiException-encoded errors. We can now return proper
OpPrereq/OpExec errors to the clients of the luxi/query socket.

The patch touches many files as we had to convert the entire call
chains in a single round. But it should be pretty straightforward
otherwise:

- change 'Result' into 'ErrorResult'
- add error annotations: change "Bad msg" into "Bad (XXXEror msg)"
- add a helper function for confd, where we don't send to client
  formatted exceptions, to convert back from ErrorResult into Result
- change tests similarly, where needed

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agoAdd exception utility functions
Iustin Pop [Fri, 12 Oct 2012 10:37:27 +0000 (12:37 +0200)]
Add exception utility functions

In Python, formatError also returns the exit code, but I find that
splitting them leads to clearer code.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agoAdd an Errors module mirroring the Python one
Iustin Pop [Sun, 7 Oct 2012 22:02:20 +0000 (00:02 +0200)]
Add an Errors module mirroring the Python one

As described in the module doc string, while writing this it dawned
upon me that we're mixing all errors together into a single hierarchy
(well, type on the Haskell side), which is not good. Some errors are
used purely within noded, some in the CLI frontends, etc. so these
should not be the same type; frontend functions should only be able to
raise frontend errors, not backend ones.

As to this patch itself, I've used again Template Haskell to generate
both the data type and the serialisation functions, as the initial
version, hand-written, seemed too prone to errors due to string
matching.

A small unittest for checking serialisation consistency is also added.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

8 years agoAbstract Luxi template functionality
Iustin Pop [Fri, 12 Oct 2012 09:09:05 +0000 (11:09 +0200)]
Abstract Luxi template functionality

These are almost generic, so let's change the signatures a bit a make
them fully so.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

8 years agoAbstract a few types in THH.hs
Iustin Pop [Fri, 12 Oct 2012 08:51:26 +0000 (10:51 +0200)]
Abstract a few types in THH.hs

This 'simple' way of defining objects will be used also for errors, so
let's make it less Luxi-specific.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agoAdd missing empty line
Guido Trotter [Wed, 24 Oct 2012 12:11:51 +0000 (14:11 +0200)]
Add missing empty line

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoRemove dumb-allocator code from devel/upload
Iustin Pop [Wed, 24 Oct 2012 10:25:30 +0000 (12:25 +0200)]
Remove dumb-allocator code from devel/upload

The 'dumb-allocator' has been removed almost two years ago (commit
6f547f96, “Remove dumb-allocator”), let's remove this special casing
for it from devel/upload.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

8 years agoFix devel/upload restart of daemons
Iustin Pop [Wed, 24 Oct 2012 10:23:25 +0000 (12:23 +0200)]
Fix devel/upload restart of daemons

While running with a wrong --prefix/--sysconfdir, I saw that
devel/upload actually uses a hardcoded path for the init script, even
though it installs it in the correct place.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

8 years agoAdd small design for Linux HA integration
Guido Trotter [Wed, 24 Oct 2012 10:08:48 +0000 (12:08 +0200)]
Add small design for Linux HA integration

This documents the status (or wanted status) of some example ocf modules
I've written for Ganeti. They are far from perfect, but they should be
shipped as a starting point for other people who want to run this to
improve on.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoReplace @LIBDIR@ in .in files
Guido Trotter [Fri, 4 Nov 2011 09:57:27 +0000 (09:57 +0000)]
Replace @LIBDIR@ in .in files

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agoImprove devel/upload command line handling
Iustin Pop [Tue, 23 Oct 2012 16:19:01 +0000 (18:19 +0200)]
Improve devel/upload command line handling

To people not used to it, it was completely non-obvious why
"./devel/upload" didn't do anything.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agoImprove logging of AssertionErrors
Iustin Pop [Tue, 23 Oct 2012 14:01:18 +0000 (16:01 +0200)]
Improve logging of AssertionErrors

Currently, when we have an assertion error raised from cmdlib, it looks like this:

  [cluster] root@node4:~# gnt-instance grow-disk instance1 0 1G
  Failure: command execution error:

This is very very confusing. This patch adds a bit of traceback
formatting to improve this as follows:

  [cluster] root@node4:~# gnt-instance grow-disk instance1 0 1G
  Failure: command execution error:
  Internal assertion error: please report this as a bug.
  Error message: ''; location:
    File "/usr/lib/python2.6/dist-packages/ganeti/cmdlib.py", line 11954, in CheckPrereq
      assert False

This is not perfect, but at least it shows better what the problem is.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agotools.prepare_node_join: Fix pep8 errors
Michael Hanselmann [Tue, 23 Oct 2012 16:01:12 +0000 (18:01 +0200)]
tools.prepare_node_join: Fix pep8 errors

Pep8 didn't agree with the indentation.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

8 years agoAdd initial implementation of prepare-node-join
Michael Hanselmann [Tue, 16 Oct 2012 14:04:15 +0000 (16:04 +0200)]
Add initial implementation of prepare-node-join

This is a new tool as per the design document “design-ssh-setup”. It
receives a JSON data structure on its standard input and configures the
SSH daemon and root's SSH keys accordingly. Unit tests are included.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agossh.GetUserFiles: RSA support, unit tests
Michael Hanselmann [Thu, 18 Oct 2012 15:34:02 +0000 (17:34 +0200)]
ssh.GetUserFiles: RSA support, unit tests

This patch changes “ssh.GetUserFiles” to support two different kinds of
SSH keys, RSA and DSA. Before it would always use DSA. Newly written
unit tests are included.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoFix typo in walkthrough document
Michele Tartara [Tue, 23 Oct 2012 07:41:14 +0000 (09:41 +0200)]
Fix typo in walkthrough document

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoFind coverage during configure, issue with Debian package
Michael Hanselmann [Mon, 22 Oct 2012 15:05:16 +0000 (17:05 +0200)]
Find coverage during configure, issue with Debian package

- Debian Squeeze and up have a package named “python-coverage”, but it
  doesn't use the same binary name as upstream (“coverage”).
- Said package includes a patch to use symlinks instead of file copies
  for jQuery. If files from previous runs are around, an exception is
  raised. This is fixed by removing all regular files and symlinks
  before generating a report.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoRemove multiple uses of '.&&.' with conjoin
Iustin Pop [Mon, 22 Oct 2012 12:19:42 +0000 (14:19 +0200)]
Remove multiple uses of '.&&.' with conjoin

This is just a bit of cleanup. The (.&&.) operator is internally just:

  a .&& b = conjoin [a, b]

so let's replace 'a .&&. b .&&. c .&&. d' directly with 'conjoin [a,
b, c, d]'.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

8 years agoImprove message for (==?) operator
Iustin Pop [Mon, 22 Oct 2012 12:07:06 +0000 (14:07 +0200)]
Improve message for (==?) operator

After seeing how nice HUnit formats the error message on failed
'assertEqual', I think we can do better with ==?. Currently it says
(on one line): "Expected equality, but 1 /= 2".

This patch changes the code to format it similar to HUnit:

  Expected equality, but got mismatch
  expected: 1
   but got: 2

(on three lines). This makes it more clear what is the expected and
what is the wrong value.

A few tests have been modified to ensure that the expected value is
the second argument to ==?. This is different than HUnit, but makes
more sense in the operator version (I think).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

8 years agoAdd a new 'really-all' make target
Iustin Pop [Mon, 22 Oct 2012 11:46:03 +0000 (13:46 +0200)]
Add a new 'really-all' make target

This is used when one wants to build all binaries, including those
that are used only for testing. A handy shortcut to make sure all
binaries can be built.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agoAnnotated inequality operator for unit test properties
Helga Velroyen [Mon, 22 Oct 2012 11:08:46 +0000 (13:08 +0200)]
Annotated inequality operator for unit test properties

This includes:
 * The operator (/=?), which checks for inequality and prints
   an error message if it encounters equality. (Basically the
   negation of the (==?) operator).
 * Application of this operator in the test property
   prop_addPri_NoN1Fail.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoFix setting of 'failN1' flag for corner case
Helga Velroyen [Mon, 22 Oct 2012 09:19:48 +0000 (11:19 +0200)]
Fix setting of 'failN1' flag for corner case

This patch includes:

* The 'failN1' flag is now only set if there is strictly less
  memory available than required for failover.
* Unit tests for that.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoA few unittests improvements
Helga Velroyen [Mon, 22 Oct 2012 09:28:22 +0000 (11:28 +0200)]
A few unittests improvements

Small simplifications of other unit tests using the (==?)
operator when possible, and typo fixes.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoCorrected some commandlines in walkthrough
Helga Velroyen [Fri, 19 Oct 2012 14:56:48 +0000 (16:56 +0200)]
Corrected some commandlines in walkthrough

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoBasicTypes.hs: fix docstring for eitherToResult
Dato Simó [Fri, 19 Oct 2012 14:54:54 +0000 (15:54 +0100)]
BasicTypes.hs: fix docstring for eitherToResult

eitherToResult now converts from `Either a b` to `GenericResult`, not
necessarily from `Either String` only. Also, fix a typo.

Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

8 years agoAdd note about running individual test cases
Iustin Pop [Fri, 19 Oct 2012 14:25:23 +0000 (16:25 +0200)]
Add note about running individual test cases

This was asked a few times offline, so let's document it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agoAdd a small QA check in instance renames for tags update
Iustin Pop [Fri, 19 Oct 2012 11:42:35 +0000 (13:42 +0200)]
Add a small QA check in instance renames for tags update

We also need to change the signature for _GetInstanceInfo, since we
don't have access to the instance dictionary in the rename instance
test.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agoUpdate blockdev's "info" at instance rename
Iustin Pop [Thu, 18 Oct 2012 15:59:56 +0000 (17:59 +0200)]
Update blockdev's "info" at instance rename

Currently, we set "info" metadata on block devices at device creation
time, but we never update it, leading to stale data in case of
instance renames. This would not be a big problem in case of regular
renames (assuming this is a rare operation), but importing instances
into the cluster via the import/export feature usually is done with a
rename; this means that all imported instances have wrong information
in their block devices.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agoLVM: remove old tags when adding new ones
Iustin Pop [Thu, 18 Oct 2012 15:58:45 +0000 (17:58 +0200)]
LVM: remove old tags when adding new ones

This patch adds a small helper function to clear an LV's tags, and
calls it at SetInfo time. We need this to be able to correctly track
instance renames, once we will call SetInfo at such times.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agoAdd a small bdev helper function
Iustin Pop [Thu, 18 Oct 2012 15:56:48 +0000 (17:56 +0200)]
Add a small bdev helper function

I wanted to write that snippet the third time, which is too much :)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

8 years agoAdd a small note about tab completion and generated files
Iustin Pop [Thu, 18 Oct 2012 14:11:13 +0000 (16:11 +0200)]
Add a small note about tab completion and generated files

This can indeed be annoying, so let's document it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

8 years agoUpdate for SSH design: Cluster name, ignore node daemon
Michael Hanselmann [Thu, 18 Oct 2012 11:38:40 +0000 (13:38 +0200)]
Update for SSH design: Cluster name, ignore node daemon

While writing the initial version of this design I misunderstood what
“setup-ssh” does. It doesn't actually start the node daemon and just
configures SSH, nothing else.

This patch adds the cluster name as a field to the JSON structure and
clarifies the purpose of the node daemon certificate given.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>