libceph: replace connection state bits with states

Use a simple set of 6 enumerated values for the socket states (CON_STATE_*)
and use those instead of the state bits.  All of the con->state checks are
now under the protection of the con mutex, so this is safe.  It also
simplifies many of the state checks because we can check for anything other
than the expected state instead of various bits for races we can think of.

This appears to hold up well to stress testing both with and without socket
failure injection on the server side.

Signed-off-by: Sage Weil <sage@inktank.com>

include/linux/ceph/messenger.h

@@ -116,18 +116,6 @@ struct ceph_msg_pos {
 #define SOCK_CLOSED	11 /* socket state changed to closed */
 #define BACKOFF         15
 
-/*
- * ceph_connection states
- */
-#define CONNECTING	1
-#define NEGOTIATING	2
-#define CONNECTED	5
-#define STANDBY		8  /* no outgoing messages, socket closed.  we keep
-			    * the ceph_connection around to maintain shared
-			    * state with the peer. */
-#define CLOSED		10 /* we've closed the connection */
-#define OPENING         13 /* open connection w/ (possibly new) peer */
-
 /*
  * A single connection with another host.
  *