Authnode

Authnode provides a general authentication & authorization service among CubeFS nodes. Client, Master, Meta and Data node are required to be authenticated and authorized before any resource access in another node.

Initially, each node (Auth, Client, Master, Meta or Data node) is launched with a secure key which is distributed by a authenticated person (for instance, cluster admin). With a valid key, a node can be identified in Authnode service and granted a ticket for resource access.

The overall work flow is: key creation and distribution –> ticket retrieval with key –> resource access with ticket.

Concepts

• Key: a bit of secret shared data between a node and Authnode that asserts identity of a client.

• Ticket: a bit of data that cryptographically asserts identity and authorization (through a list of capabilities) for a service for a period of time.

• Capability: a capability is defined in the format of node:object:action where node refers to a service node (such as auth, master, meta or data), object refers to the resource and action refers to permitted activities (such as read, write and access). See examples below.

Capability Example

Capability	Specifications
auth:createkey:access	Access permission for createkey in Authnode
master:*:*	Any permissions for any objects in Master node
*:*:*	Any permissions for any objects in any nodes

Client Tool

cfs-authtool is a client-side utility of Authnode, providing key managements (creation, view and modification) and ticket retrieval in and from Authnode keystore. In particular, Each key is associated with an entity name or id, secret key string, creation time, role and capability specification.

Key structure

Key	Type	Description
id	string	Unique key identifier composed of letters and digits
key	string	Base64 encoded secret key
role	string	The role of the key (either client or service)
caps	string	The capabilities of the key

Build

Use the following commands to build client side tool for Authnode:

$ git clone http://github.com/cubefs/cubefs.git
$ cd cubefs
$ make build

If successful, the tool cfs-authtool can be found in build/bin.

Synopsis

cfs-authtool ticket -host=AuthNodeAddress [-keyfile=Keyfile] [-output=TicketOutput] [-https=TrueOrFalse -certfile=AuthNodeCertfile] TicketService Service

cfs-authtool api -host=AuthNodeAddress -ticketfile=TicketFile [-data=RequestDataFile] [-output=KeyFile] [-https=TrueOrFalse -certfile=AuthNodeCertfile] Service Request

cfs-authtool authkey [-keylen=KeyLength]

TicketService := [getticket]

Service := [AuthService | MasterService | MetaService | DataService]

Request := [createkey | deletekey | getkey | addcaps | deletecaps | getcaps | addraftnode | removeraftnode]

AuthNode Configurations

Authnode use JSON as configuration file format.

Properties

Key	Type	Description	Mandatory
role	string	Role of process and must be set to master	Yes
ip	string	host ip	Yes
port	string	Http port which api service listen on	Yes
prof	string	golang pprof port	Yes
id	string	identy different master node	Yes
peers	string	the member information of raft group	Yes
logDir	string	Path for log file storage	Yes
logLevel	string	Level operation for logging. Default is error.	No
retainLogs	string	the number of raft logs will be retain.	Yes
walDir	string	Path for raft log file storage.	Yes
storeDir	string	Path for RocksDB file storage,path must be exist	Yes
clusterName	string	The cluster identifier	Yes
exporterPort	int	The prometheus exporter port	No
authServiceKey	string	The secret key used for authentication of AuthNode	Yes
authRootKey	string	The secret key used for key derivation (session and client secret key)	Yes
enableHTTPS	bool	Option whether enable HTTPS protocol	No

Example:

{
  "role": "authnode",
  "ip": "192.168.0.14",
  "port": "8080",
  "prof":"10088",
  "id":"1",
  "peers": "1:192.168.0.14:8080,2:192.168.0.15:8081,3:192.168.0.16:8082",
  "logDir": "/export/Logs/authnode",
  "logLevel":"info",
  "retainLogs":"100",
  "walDir":"/export/Data/authnode/raft",
  "storeDir":"/export/Data/authnode/rocksdbstore",
  "exporterPort": 9510,
  "consulAddr": "http://consul.prometheus-cfs.local",
  "clusterName":"test",
  "authServiceKey":"9h/sNq4+5CUAyCnAZM927Y/gubgmSixh5hpsYQzZG20=",
  "authRootKey":"wbpvIcHT/bLxLNZhfo5IhuNtdnw1n8kom+TimS2jpzs=",
  "enableHTTPS":false
}

Steps for Starting CubeFS with AuthNode

Create Authnode key

Run the command:

$ ./cfs-authtool authkey

If successful, two key files can be generated authroot.json and authservice.json under current directory. They represent authServiceKey and authRootKey respectively.

example authservice.json :

{
     "id": "AuthService",
     "key": "9h/sNq4+5CUAyCnAZM927Y/gubgmSixh5hpsYQzZG20=",
     "create_ts": 1573801212,
     "role": "AuthService",
     "caps": "{\"*\"}"
}

Edit authnode.json in docker/conf as following:

• authRootKey: use the value of key in authroot.json

• authServiceKey: use the value of key in authService.json

example authnode.json :

{
     "role": "authnode",
     "ip": "192.168.0.14",
     "port": "8080",
     "prof":"10088",
     "id":"1",
     "peers": "1:192.168.0.14:8080,2:192.168.0.15:8081,3:192.168.0.16:8082",
     "retainLogs":"2",
     "logDir": "/export/Logs/authnode",
     "logLevel":"info",
     "walDir":"/export/Data/authnode/raft",
     "storeDir":"/export/Data/authnode/rocksdbstore",
     "exporterPort": 9510,
     "consulAddr": "http://consul.prometheus-cfs.local",
     "clusterName":"test",
     "authServiceKey":"9h/sNq4+5CUAyCnAZM927Y/gubgmSixh5hpsYQzZG20=",
     "authRootKey":"wbpvIcHT/bLxLNZhfo5IhuNtdnw1n8kom+TimS2jpzs=",
     "enableHTTPS":false
}

Start Authnode Cluster

In directory docker/authnode, run the following command to start a Authnode cluster.

$ docker-compose up -d

Create admin in Authnode

Get Authnode ticket using authServiceKey:

$ ./cfs-authtool ticket -host=192.168.0.14:8080 -keyfile=authservice.json -output=ticket_auth.json getticket AuthService

example ticket_auth.json :

{
    "id": "AuthService",
    "session_key": "A9CSOGEN9CFYhnFnGwSMd4WFDBVbGmRNjaqGOhOinJE=",
    "service_id": "AuthService",
    "ticket": "RDzEiRLX1xjoUyp2TDFviE/eQzXGlPO83siNJ3QguUrtpwiHIA3PLv4edyKzZdKcEb3wikni8UxBoIJRhKzS00+nB7/9CjRToAJdT9Glhr24RyzoN8psBAk82KEDWJhnl+Y785Av3f8CkNpKv+kvNjYVnNKxs7f3x+Ze7glCPlQjyGSxqARyLisoXoXbiE6gXR1KRT44u7ENKcUjWZ2ZqKEBML9U4h0o58d3IWT+n4atWKtfaIdp6zBIqnInq0iUueRzrRlFEhzyrvi0vErw+iU8w3oPXgTi+um/PpUyto20c1NQ3XbnkWZb/1ccx4U0"
}

Create admin using Authnode ticket:

$ ./cfs-authtool api -host=192.168.0.14:8080 -ticketfile=ticket_auth.json -data=data_admin.json -output=key_admin.json AuthService createkey

example data_admin.json :

{
    "id": "admin",
    "role": "service",
    "caps": "{\"API\":[\"*:*:*\"]}"
}

Create key for CubeFS cluster

• Get Authnode ticket using admin key:

$ ./cfs-authtool ticket -host=192.168.0.14:8080 -keyfile=key_admin.json -output=ticket_admin.json getticket AuthService

• Create key for Master

$ ./cfs-authtool api -host=192.168.0.14:8080 -ticketfile=ticket_admin.json -data=data_master.json -output=key_master.json AuthService createkey

example data_master.json :

{
    "id": "MasterService",
    "role": "service",
    "caps": "{\"API\":[\"*:*:*\"]}"
}

Specifications:

Key	Description
id	will set Client ID
role	will set the role of id
caps	will set the capabilities of id

Edit master.json as following:

• masterServiceKey: use the value of key in key_master.json

Create key for Client

$ ./cfs-authtool api -host=192.168.0.14:8080 -ticketfile=ticket_admin.json -data=data_client.json -output=key_client.json AuthService createkey

example data_client.json:

{
    "id": "ltptest",
    "role": "client",
    "caps": "{\"API\":[\"*:*:*\"], \"Vol\":[\"*:*:*\"]}"
}

Edit client.json as following:

• clientKey: use the value of key in key_client.json

example client.json：

{
    "masterAddr": "192.168.0.11:17010,192.168.0.12:17010,192.168.0.13:17010",
    "mountPoint": "/cfs/mnt",
    "volName": "ltptest",
    "owner": "ltptest",
    "logDir": "/cfs/log",
    "logLevel": "info",
    "consulAddr": "http://192.168.0.101:8500",
    "exporterPort": 9500,
    "profPort": "17410",
    "authenticate": true,
    "ticketHost": "192.168.0.14:8080,192.168.0.15:8081,192.168.0.16:8082",
    "clientKey": "jgBGSNQp6mLbu7snU8wKIdEkytzl+pO5/OZOJPpIgH4=",
    "enableHTTPS": "false"
}

Specifications:

Key	Description
authenticate	will enable authentication flow if set true
ticketHost	will set the IP/URL of Authnode cluster
clientKey	will set the key generated by Authnode
enableHTTPS	will enable HTTPS if set true

Start CubeFS cluster

Run the following to launch CubeFS cluster with AuthNode enabled:

$ docker/run_docker.sh -r -d /data/disk

Generate Certificate

To prevent MITM (Man In The Middle) attacks, HTTPS is required for the communication between client and service. The following steps show the generation of self-sign a certificate with a private (.key) and public key.

Generating Key and Self Signed Cert:

$ openssl req \
  -x509 \
  -nodes \
  -newkey rsa:2048 \
  -keyout server.key \
  -out server.crt \
  -days 3650 \
  -subj "/C=GB/ST=China/L=Beijing/O=jd.com/OU=Infra/CN=*"

Specifications:

• server.crt: AuthNode public certificate needed to be sent to Client

• server.key: AuthNode private key needed to be securely placed in /app folder in Authnode

For easy deployment, current implementation of AuthNode uses TLS option insecure_skip_verify and tls.RequireAndVerifyClientCert, which would skip secure verification of both client and server. For environment with high security command, these options should be turned off.

Master API Authentication

Master has numerous APIs, such as creating and deleting volumes, so it is necessary to authenticate access to the master API to improve cluster security.

With the excellent authentication capability of authnode, we have optimized the existing authentication mechanism to achieve the goal of simplifying the authentication process.

Enable Master API Authentication

example master.json ：

{
  "role": "master",
  "ip": "127.0.0.1",
  "listen": "17010",
  "prof":"17020",
  "id":"1",
  "peers": "1:127.0.0.1:17010,2:127.0.0.2:17010,3:127.0.0.3:17010",
  "retainLogs":"20000",
  "logDir": "/cfs/master/log",
  "logLevel":"info",
  "walDir":"/cfs/master/data/wal",
  "storeDir":"/cfs/master/data/store",
  "consulAddr": "http://consul.prometheus-cfs.local",
  "clusterName":"cubefs01",
  "metaNodeReservedMem": "1073741824",
  "masterServiceKey": "jgBGSNQp6mLbu7snU8wKIdEkytzl+pO5/OZOJPpIgH4=",
  "authenticate": true,
  "authNodeHost": "192.168.0.14:8080,192.168.0.15:8081,192.168.0.16:8082",
  "authNodeEnableHTTPS": false
}

Properties

Key	Description
authenticate	will enable Master API Authentication if set true
authNodeHost	will set the IP/URL of Authnode cluster
authNodeEnableHTTPS	will enable HTTPS if set true

Use Authentication Parameter

When accessing the Master API, the parameter clientIDKey for authentication must be included.

When using authtool to create a key, auth_id_key is generated. This key will be used as the clientIDKey when accessing the master API.

First Example

Access the master API via HTTP and write the parameter clientIDKey, such as expanding a volume:

curl --location 'http://127.0.0.1:17010/vol/update?name=ltptest&authKey=0e20229116d5a9a4a9e876806b514a85&capacity=100&clientIDKey=eyJpZCI6Imx0cHRlc3QiLCJhdXRoX2tleSI6ImpnQkdTTlFwNm1MYnU3c25VOHdLSWRFa3l0emwrcE81L09aT0pQcElnSDQ9In0='

Second Example

Access the master API via cfs-cli and write the clientIDKey to configuration file .cfs-cli.json, so that any cluster management command is authenticated for permissions.

example .cfs-cli.json ：

{
  "masterAddr": [
    "127.0.0.1:17010",
    "127.0.0.2:17010",
    "127.0.0.3:17010"
  ],
  "timeout": 60,
  "clientIDKey": "eyJpZCI6Imx0cHRlc3QiLCJhdXRoX2tleSI6ImpnQkdTTlFwNm1MYnU3c25VOHdLSWRFa3l0emwrcE81L09aT0pQcElnSDQ9In0="
}

Third Example

When datanode and metanode are started, they will respectively call the AddDataNode and AddMetaNode APIs, so it is also necessary to prepare serviceIDKey for them.

Similarly, use authtool to create keys for datanode and metanode respectively, and write the key as the serviceIDKey to the configuration file. When they start, permission authentication will be performed.

Create Key For datanode

$ ./cfs-authtool api -host=192.168.0.14:8080 -ticketfile=ticket_admin.json -data=data_datanode.json -output=key_datanode.json AuthService createkey

example data_datanode ：

{
    "id": "DatanodeService",
    "role": "service",
    "caps": "{\"API\":[\"*:*:*\"]}"
}

Edit datanode.json as following:

• serviceIDKey: use the value of auth_id_key in key_datanode.json

Create Key For metanode

$ ./cfs-authtool api -host=192.168.0.14:8080 -ticketfile=ticket_admin.json -data=data_metanode.json -output=key_metanode.json AuthService createkey

example data_metanode ：

{
    "id": "MetanodeService",
    "role": "service",
    "caps": "{\"API\":[\"*:*:*\"]}"
}

Edit metanode.json as following:

• serviceIDKey: use the value of auth_id_key in key_metanode.json

Edit on GitHub

Using Erasure Coding System GUI