Access Configuration

Access is the access module, mainly responsible for data upload, download, deletion, etc.

Access configuration is based on the public configuration, and the following configuration instructions mainly apply to private configuration for Access.

Configuration Instructions

First-Level Configuration

Configuration ItemDescriptionRequired
Public Configuration ItemsRefer to the Basic Service Configuration sectionYes
service_registerService registration informationYes, can be used for service discovery in Access after configuration
limitRate limiting configurationNo, single-machine rate limiting configuration
streamMain Access configuration itemYes, refer to the following second-level configuration options

Second-Level Stream Configuration

Configuration ItemDescriptionRequired
idcIDC for the serviceYes
max_blob_sizeFile segment blob sizeNo, default is 4MB
mem_pool_size_classesMemory control for file read/writeNo
encoder_concurrencyEC encoding/decoding concurrencyNo, default is 1000
encoder_enableverifyWhether to enable EC encoding/decoding verificationNo, default is enabled
min_read_shards_xNumber of shards to download concurrently for EC readingNo, default is 1. The larger the number, the higher the fault tolerance, but also the higher the bandwidth.
shard_crc_disabledWhether to verify the data CRC of the blobnodeNo, default is enabled
disk_punish_interval_sInterval for temporarily marking a bad diskNo, default is 60s
service_punish_interval_sInterval for temporarily marking a bad serviceNo, default is 60s
blobnode_configBlobnode RPC configurationRefer to the RPC configuration section rpc
proxy_configProxy RPC configurationRefer to the RPC configuration section rpc
cluster_configMain cluster configurationYes, refer to the following third-level configuration options

Third-Level Cluster Configuration

Configuration ItemDescriptionRequired
regionRegion informationYes, do not change after configuration
region_magicCRC field used for encoding file locationYes, do not change after configuration. If changed, all locations will be invalidated.
consul_agent_addrConsul address for cluster informationYes
cluster_reload_secsInterval for synchronizing cluster informationNo, default is 3s
service_reload_secsInterval for synchronizing service informationNo, default is 3s
clustermgr_client_configClustermgr RPC configurationRefer to the RPC configuration example rpc

Configuration Example

service_register

Note

Support for health_port began with version v3.2.1.

  • consul_addr: Consul address for Access service registration
  • service_ip: Access service bind IP
  • node: Hostname
  • health_port: Health check port range for Consul
{
    "consul_addr": "127.0.0.1:8500",
    "service_ip": "127.0.0.1",
    "node": "access-node1",
    "health_port": [9700, 9799]
}

limit

  • reader_mbps: Single-machine download bandwidth (MB/s)
  • writer_mbps: Single-machine upload bandwidth (MB/s)
  • name_rps: RPS limit for each interface
{
    "name_rps": {
        "alloc": 0,
        "put": 100,
        "putat": 0,
        "get": 0,
        "delete": 0,
        "sign": 0
    },
    "reader_mbps": 1000,
    "writer_mbps": 200
}

mem_pool_size_classes

  • key: Memory allocation ladder
  • value: Limit on the number of items, 0 means no limit (Access currently does not enable quantity limits)
{
    "2048": 0,
    "65536": 0,
    "524288": 0,
    "2097152": 10240,
    "8389632": 4096,
    "16777216": 1024,
    "33554432": 512,
    "67108864": 64
}

Complete Example

{
    "max_procs": 0,
    "shutdown_timeout_s": 30,
    "log": {
        "level": "info",
        "filename": "./run/logs/access.log",
        "maxsize": 1024,
        "maxage": 7,
        "maxbackups": 7
    },
    "bind_addr": ":9500",
    "service_register": {
        "consul_addr": "127.0.0.1:8500",
        "service_ip": "127.0.0.1",
        "node": "access-node1",
        "health_port": [9700, 9799]
    },
    "limit": {
        "name_rps": {
           "put": 100
        },
        "reader_mbps": 1000,
        "writer_mbps": 200
    },
    "stream": {
        "idc": "idc",
        "max_blob_size": 4194304,
        "mem_pool_size_classes": {
            "2048": 0,
            "65536": 0,
            "524288": 0,
            "2097152": 10240,
            "8389632": 4096,
            "16777216": 1024,
            "33554432": 512,
            "67108864": 64
        },
        "encoder_concurrency": 1000,
        "encoder_enableverify": true,
        "min_read_shards_x": 1,
        "shard_crc_disabled": false,
        "cluster_config": {
            "region": "region",
            "region_magic": "region",
            "cluster_reload_secs": 3,
            "service_reload_secs": 3,
            "clustermgr_client_config": {
                "client_timeout_ms": 3000,
                "transport_config": {
                    "auth": {
                        "enable_auth": true,
                        "secret": "secret key"
                    },
                    "dial_timeout_ms": 2000
                }
            },
            "consul_agent_addr": "127.0.0.1:8500"
        },
        "disk_punish_interval_s": 60,
        "service_punish_interval_s": 60,
        "blobnode_config": {
            "client_timeout_ms": 10000
        },
        "proxy_config": {
            "client_timeout_ms": 5000
        }
    }
}
Edit on GitHub