How to debug if "Failed to connect to aggregator"

[zjamy]

Hi there,
How to debug if I met below error log? Thanks

(07-27) 15:52:04 INFO     [executor.py:113] Data partitioner starts ...
(07-27) 15:52:04 INFO     [divide_data.py:106] Randomly partitioning data, 50000 samples...
(07-27) 15:52:04 INFO     [divide_data.py:106] Randomly partitioning data, 10000 samples...
(07-27) 15:52:04 INFO     [executor.py:124] Data partitioner completes ...
(07-27) 15:52:04 INFO     [channel_context.py:20] %%%%%%%%%% Opening grpc connection to 127.0.0.1 %%%%%%%%%%
(07-27) 15:52:04 INFO     [executor.py:412] Start monitoring events ...
(07-27) 15:52:07 WARNING  [executor.py:397] Failed to connect to aggregator <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1658937127.191636762","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3260,"referenced_errors":[{"created":"@1658937127.191635791","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":167,"grpc_status":14}]}"
>. Will retry in 5 sec.

[AmberLJC]

which dataset you are using? And can you share the config file and your command as well?

[fanlai0990]

Yes. Because we need to start aggregator first, and then the workers. Or you can use fedscale driver submit PATH_OF_CONF. Please remember to kill jobs via fedscale driver stop LAST_JOB_NAME.

[zjamy]

I re-installed everything, and still met the same issue.
Here is the command:

!bash fedscale.sh driver submit benchmark/configs/dry_run/dry_run.yml

Here is the error log of "dryrun_logging"

start to work
(07-28) 22:02:19 INFO     [aggregator.py:28] Job args Namespace(adam_epsilon=1e-08, arrival_interval=3, async_buffer=10, async_mode=False, backbone='./resnet50.pth', backend='gloo', batch_size=32, bidirectional=True, blacklist_max_len=0.3, blacklist_rounds=-1, block_size=64, cfg_file='./utils/rcnn/cfgs/res101.yml', checkin_period=50, clf_block_size=32, clip_bound=0.9, clip_threshold=3.0, clock_factor=2.4368231046931412, conf_path='~/dataset/', connection_timeout=60, cuda_device=None, cut_off_util=0.05, data_cache='', data_dir='FedScale/benchmark/dataset/data/', data_map_file=None, data_set='cifar10', decay_factor=0.98, decay_round=10, device_avail_file=None, device_conf_file='/tmp/client.cfg', dump_epoch=10000000000.0, embedding_file='glove.840B.300d.txt', engine='pytorch', epsilon=0.9, eval_interval=5, executor_configs='127.0.0.1:[0,4,4,4]', experiment_mode='simulation', exploration_alpha=0.3, exploration_decay=0.98, exploration_factor=0.9, exploration_min=0.3, filter_less=0, filter_more=1000000000000000.0, finetune=False, gamma=0.9, gradient_policy=None, hidden_layers=7, hidden_size=256, input_dim=0, job_name='dryrun', labels_path='labels.json', learning_rate=0.001, line_by_line=False, local_steps=20, log_path='FedScale/benchmark', loss_decay=0.2, malicious_factor=1000000000000000.0, memory_capacity=2000, min_learning_rate=5e-05, mlm=False, mlm_probability=0.15, model='resnet18', model_size=65536, model_zoo='torchcv', n_actions=2, n_states=4, noise_dir=None, noise_factor=0.1, noise_max=0.5, noise_min=0.0, noise_prob=0.4, num_class=10, num_classes=35, num_executors=12, num_loaders=2, num_participants=4, output_dim=0, overcommitment=1.3, overwrite_cache=False, pacer_delta=5, pacer_step=20, proxy_mu=0.1, ps_ip='127.0.0.1', ps_port='39889', rnn_type='lstm', round_penalty=2.0, round_threshold=30, rounds=200, sample_mode='random', sample_rate=16000, sample_seed=233, sample_window=5.0, spec_augment=False, speed_volume_perturb=False, target_delta=0.0001, target_replace_iter=15, task='cv', test_bsz=32, test_manifest='data/test_manifest.csv', test_output_dir='./logs/server', test_ratio=1.0, test_size_file='', this_rank=0, time_stamp='0728_220218', train_manifest='data/train_manifest.csv', train_size_file='', train_uniform=False, upload_step=20, use_cuda=True, vocab_tag_size=500, vocab_token_size=10000, weight_decay=0, window='hamming', window_size=0.02, window_stride=0.01, yogi_beta=0.9, yogi_beta2=0.99, yogi_eta=0.003, yogi_tau=1e-08)
2022-07-28 22:02:19.874038: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
(07-28) 22:02:21 INFO     [aggregator.py:124] Initiating control plane communication ...
(07-28) 22:02:21 INFO     [aggregator.py:148] %%%%%%%%%% Opening aggregator sever using port [::]:39889 %%%%%%%%%%
(07-28) 22:02:21 INFO     [fllibs.py:105] Initializing the model ...
(07-28) 22:02:21 INFO     [aggregator.py:855] Start monitoring events ...
start to work
(07-28) 22:02:31 INFO     [executor.py:57] (EXECUTOR:1) is setting up environ ...
(07-28) 22:02:31 INFO     [fllibs.py:105] Initializing the model ...
start to work
(07-28) 22:02:33 INFO     [executor.py:57] (EXECUTOR:2) is setting up environ ...
(07-28) 22:02:33 INFO     [fllibs.py:105] Initializing the model ...
start to work
(07-28) 22:02:35 INFO     [executor.py:113] Data partitioner starts ...
(07-28) 22:02:35 INFO     [divide_data.py:106] Randomly partitioning data, 50000 samples...
(07-28) 22:02:35 INFO     [divide_data.py:106] Randomly partitioning data, 10000 samples...
(07-28) 22:02:35 INFO     [executor.py:124] Data partitioner completes ...
(07-28) 22:02:35 INFO     [channel_context.py:20] %%%%%%%%%% Opening grpc connection to 127.0.0.1 %%%%%%%%%%
(07-28) 22:02:35 INFO     [executor.py:412] Start monitoring events ...
(07-28) 22:02:35 WARNING  [executor.py:397] Failed to connect to aggregator <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1659016955.388347919","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3186,"referenced_errors":[{"created":"@1659016955.388347506","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":146,"grpc_status":14}]}"
>. Will retry in 5 sec.

[zjamy]

some grpc related debug is required?

[zjamy]

I think I work-arounded by changing "127.0.0.1" to "localhost".

I tried to run a simple grpc helloword python, and found that if I use 127.0.0.1, I will meet the same issue.

Thank you.

[fanlai0990]

Great to hear this! We would greatly appreciate it if you can consider submitting your Pull Request if you have any interesting FL work or find any bugs. Thank you!